Self-attention is a core idea behind modern language models because it lets each word in a sequence look at other words and decide which ones matter most. Instead of processing words only in order, the model builds relationships across the whole sentence at once. This helps it capture context, resolve ambiguity, and represent meaning more effectively.
Self-attention is one reason transformer models work so well in translation, chatbots, summarization, and code generation.
The mechanism works by turning each input token into three vectors called query, key, and value. A token compares its query to the keys of all tokens to produce attention scores, then those scores are normalized with softmax into weights that sum to 1. The final output for that token is a weighted sum of the value vectors, so important tokens contribute more strongly.
Repeating this process across all tokens allows the model to build context-aware representations in parallel.
Understanding Self-Attention Mechanism
The three vector types have different jobs because comparison and information transfer are not the same task. A query can be thought of as a pattern a token is seeking. A key is the pattern that token offers for matching.
A value is the information that may be passed onward if the match is useful. These vectors come from learned weight tables. At the start of training, their matches are mostly meaningless.
By adjusting the weights after many examples, the model learns useful patterns such as links between a pronoun and a noun, a verb and its subject, or a word and a nearby negation. The same input word can produce different attention results when its surrounding words change.
In practice, the calculation is usually done with matrices rather than one pair of vectors at a time. One matrix holds the queries for every token. Another holds all keys.
Multiplying these matrices creates a grid of scores. Each row belongs to one token that is looking for relevant information. Each column shows how strongly it considered a possible source token.
Before softmax, the scores are divided by the square root of the key vector size. Dot products tend to grow larger when vectors have many components.
Without scaling, softmax can become too extreme, placing nearly all weight on one position. That makes learning less stable because tiny score changes can cause large weight changes.
Attention does not automatically understand grammar or meaning. It learns statistical relationships from training data. A high attention weight can sometimes point to a word that appears relevant to people, but it is not a complete explanation of the model's decision.
Information is transformed many times through value projections, output layers, feed forward networks, and later attention layers. Multi-head attention helps because one head may focus on local word order while another tracks a long distance reference.
Some heads may become less useful than others. Their results are combined so later layers can use several kinds of context at once.
A crucial detail is masking. In tasks that generate text one token at a time, a token must not read tokens that come later in the sentence. A causal mask blocks those future positions before softmax, usually by giving them an extremely low score.
This prevents the model from seeing the answer during training. Other tasks, such as classifying a full sentence, can allow every token to use both earlier and later context. Padding masks handle blank positions added to make sequences equal length in a batch.
When studying attention, track the dimensions at every stage and remember which direction each row represents. Check whether a mask is present, whether scores are scaled, and whether weights sum to one across the correct row. These details prevent many common mistakes.
Key Facts
- Each token is projected into query, key, and value vectors: Q = XW_Q, K = XW_K, V = XW_V
- Raw attention scores are computed with a dot product: score(i,j) = q_i · k_j
- Scaled dot-product attention uses: Attention(Q,K,V) = softmax(QK^T / sqrt(d_k))V
- The softmax step converts scores into weights that sum to 1 across compared tokens
- A token's output is a weighted sum of value vectors: output_i = sum_j a_ij v_j
- Multi-head attention runs several attention operations in parallel, then combines them: MultiHead = Concat(head_1,...,head_h)W_O
Vocabulary
- Token
- A token is a basic unit of input, such as a word, subword, or symbol, that the model processes.
- Query vector
- A query vector represents what information a token is looking for from other tokens.
- Key vector
- A key vector represents what kind of information a token offers to other tokens.
- Value vector
- A value vector contains the information that gets combined into the final attention output.
- Softmax
- Softmax is a function that turns a list of scores into positive weights that add up to 1.
Common Mistakes to Avoid
- Assuming attention scores are the final output, which is wrong because the scores must first be normalized into weights and then applied to the value vectors.
- Forgetting the scaling factor 1/sqrt(d_k), which is wrong because large dot products can make softmax too sharp and hurt training stability.
- Thinking self-attention only compares neighboring words, which is wrong because each token can attend to every token in the sequence unless masking limits it.
- Mixing up keys and values, which is wrong because keys are used for matching with queries while values are the vectors actually combined to form the output.
Practice Questions
- 1 A token has query q = [1, 2]. Two tokens have keys k1 = [1, 0] and k2 = [0, 2]. Compute the raw attention scores q · k1 and q · k2.
- 2 Suppose a token has attention weights [0.25, 0.75] over two value vectors v1 = 4 and v2 = 10. Compute the weighted output.
- 3 Explain why self-attention can represent the meaning of the word bank differently in the sentences I sat by the bank and I went to the bank to deposit money.