Attention in AI is a method that helps a model decide which parts of the input are most important for a task. It is used in tools that translate languages, answer questions, summarize text, generate images, and understand speech. Instead of treating every word or data point as equally important, attention gives stronger weight to the parts that matter most.
This makes many modern AI systems better at handling long sentences, large data sets, and complex patterns.
In a transformer model, each token, such as a word or word piece, is converted into numbers and compared with other tokens. The model creates attention scores, turns them into probabilities, and uses those probabilities to mix information from the most relevant tokens. This process is closely connected to statistics because the model uses weighted averages and probability-like values.
It is also connected to computer science because the steps can be represented with code, matrices, tables, and graph-style visualizations.
Key Facts
- Attention assigns weights to input parts so the model can focus more on useful information.
- Attention score = query dot key, often written as score = Q · K.
- Scaled dot-product attention uses Attention(Q, K, V) = softmax(QK^T / sqrt(dk))V.
- The softmax function turns scores into weights that add up to 1.
- Output vector = sum of attention weight × value vector for each token.
- Multi-head attention lets a model look for several kinds of relationships at the same time.
Vocabulary
- Attention
- Attention is a machine learning method that gives different importance weights to different parts of the input.
- Token
- A token is a small unit of input, such as a word, part of a word, symbol, or data item.
- Query
- A query is a vector that represents what a token is looking for in other tokens.
- Key
- A key is a vector that represents what information a token can offer for comparison.
- Value
- A value is a vector containing the information that gets combined after attention weights are chosen.
Common Mistakes to Avoid
- Thinking attention is the same as human attention. AI attention is a mathematical weighting process, not awareness or understanding.
- Assuming the biggest word is always the most important word. Attention depends on the task and the surrounding context, not only on word length or position.
- Forgetting that attention weights must be normalized. Raw scores are not final importance values until a function like softmax turns them into weights that add up to 1.
- Treating one attention head as the whole model. A transformer usually uses many attention heads and other layers, so one set of weights shows only part of the process.
Practice Questions
- 1 A model gives attention weights 0.50, 0.30, and 0.20 to three value numbers 10, 4, and 8. Compute the weighted output.
- 2 Three raw attention scores are 2, 1, and 0. Using the simplified weights 0.67, 0.24, and 0.09 after softmax, find the weighted average of values 6, 3, and 12.
- 3 In the sentence The robot picked up the apple because it was ripe, explain why an attention system should connect it more strongly to apple than to robot.