Self-attention is a key idea behind many modern AI systems, including transformer models used for translation, search, chatbots, and image tools. It helps a model decide which words or data points are most important to focus on when making a prediction. Instead of reading a sentence only from left to right, self-attention lets every word compare itself with every other word.
This matters because meaning often depends on relationships between words that may be far apart.
Key Facts
- Self-attention lets each token compare itself with all other tokens in the input.
- A token can be a word, part of a word, a symbol, or another piece of data.
- Attention score = similarity(query, key), often using a dot product.
- Scaled dot-product attention uses Attention(Q, K, V) = softmax(QK^T / sqrt(dk))V.
- Softmax turns raw attention scores into weights that add to 1.
- Multi-head attention runs several attention patterns at once so the model can track different relationships.
Vocabulary
- Self-attention
- A method that lets each token in a sequence decide how strongly it should use information from the other tokens.
- Token
- A small piece of input data, such as a word, part of a word, number, or symbol, that a model processes.
- Query
- A vector that represents what a token is looking for when it compares itself with other tokens.
- Key
- A vector that represents what information a token offers for comparison with queries.
- Value
- A vector that contains the information passed forward after attention weights are applied.
Common Mistakes to Avoid
- Thinking self-attention is the same as human attention is wrong because it is a mathematical weighting process, not awareness or understanding.
- Ignoring word order is wrong because transformers usually add position information so the model can tell the difference between sentences with the same words in different orders.
- Treating the largest attention weight as the only important one is wrong because the final output is usually a weighted mix of many values.
- Forgetting to scale the dot product is wrong because large scores can make softmax too extreme and can make learning less stable.
Practice Questions
- 1 A token has attention weights 0.50, 0.30, and 0.20 for three value numbers 10, 4, and 7. What weighted output does it produce?
- 2 For a sentence with 6 tokens, each token compares with every token including itself. How many query-key comparisons are made in one self-attention layer?
- 3 In the sentence 'The robot moved the box because it was heavy,' explain why self-attention can help the model decide what the word 'it' refers to.