Transformers are a neural network architecture designed to process sequences such as text, code, and even image patches. They became famous because they handle long range relationships more effectively than many earlier sequence models like recurrent neural networks. The 2017 paper Attention Is All You Need showed that attention mechanisms alone could drive powerful language understanding and generation.
Today, transformers power chatbots, translation systems, search tools, and many modern AI applications.
A transformer converts input tokens into vectors, adds positional information, and repeatedly refines those vectors through attention and feedforward layers. Self attention lets each token weigh the importance of other tokens in the sequence, which helps the model capture context. In encoder decoder versions, the encoder builds contextual representations and the decoder uses them to generate outputs one token at a time.
Training adjusts millions or billions of parameters so the model assigns high probability to correct next tokens or target outputs.
Understanding Transformer Architecture Explained
Inside an attention layer, every token representation is turned into three learned views called a query, a key, and a value. A query represents what the current token needs from the surrounding context. A key represents what information a token can offer.
A value carries the information that may be passed on. The model compares queries with keys, then turns the comparisons into attention weights. Large weights mean that one token should borrow more information from another.
In the sentence "The trophy did not fit in the suitcase because it was too big," attention can help connect "it" with the trophy rather than the suitcase. This is useful because word meaning often depends on distant words.
Multi head attention gives the model several separate ways to inspect the same sequence. One head may learn links between a pronoun and a noun. Another may focus on nearby grammar.
A different head may notice phrase boundaries or repeated ideas. These heads do not come with built in language rules. Their patterns are learned from training examples.
Some heads become useful while others contribute little. Students should avoid thinking that every attention map is a complete explanation of a model decision. Attention shows how information is mixed at one point in the network, but later layers can change that information many times.
Each attention sublayer is followed by a small feed forward network that processes each position separately. Attention moves information between positions. The feed forward network transforms the resulting features at each position.
It usually expands the vector into a larger hidden space, applies a nonlinear activation, then reduces it again. This gives the model capacity to build complex features instead of only averaging information from other tokens. Residual connections add the earlier representation back into the updated one.
This preserves useful information and helps gradients travel through many layers during training. Layer normalization keeps values in a stable range, which makes deep networks easier to train.
The decoder needs one extra safety rule when producing text. A mask blocks each position from seeing future target tokens. Without that mask, a model trained to predict the next word could secretly read the answer later in the sentence.
In translation, the decoder can attend to the encoder output for the source sentence while its masked attention follows the words already generated. During training, the correct earlier target words are supplied so many positions can be learned at once. During use, the model has only its own previous output, so errors can build up.
This explains why generated text can begin clearly yet drift, repeat, or state unsupported claims. When studying transformer diagrams, track the direction of information, the purpose of each mask, and where a representation is copied through a residual path.
Key Facts
- Token embeddings map discrete tokens to vectors: x_i in R^d_model
- Positional encoding adds order information: z_i = x_i + p_i
- Attention scores are computed by dot products: score(i,j) = q_i · k_j
- Scaled dot product attention: Attention(Q,K,V) = softmax(QK^T / sqrt(d_k))V
- Multi head attention runs several attention operations in parallel, then concatenates them: head_h = Attention(Q_h,K_h,V_h)
- Output probabilities are produced with a linear layer and softmax: P(token) = softmax(Wy + b)
Vocabulary
- Token
- A token is a basic unit of input such as a word, subword, or symbol that the model processes.
- Embedding
- An embedding is a learned vector representation that places similar tokens closer together in a high dimensional space.
- Self attention
- Self attention is a mechanism that lets each token compare itself with other tokens in the same sequence to gather context.
- Encoder
- The encoder is the part of a transformer that converts input tokens into contextualized internal representations.
- Decoder
- The decoder is the part of a transformer that uses previous outputs and encoder information to generate the next token.
Common Mistakes to Avoid
- Thinking attention is just selecting one important word, which is wrong because attention usually distributes weights across many tokens and combines their information.
- Ignoring positional encoding, which is wrong because attention alone does not inherently know token order in a sequence.
- Assuming the encoder and decoder are always both present, which is wrong because some transformers are encoder only or decoder only depending on the task.
- Forgetting the scaling factor sqrt(d_k) in attention, which is wrong because large dot products can make the softmax too sharp and hurt training stability.
Practice Questions
- 1 A model uses d_k = 64. In scaled dot product attention, by what number do you divide each entry of QK^T before applying softmax?
- 2 A sentence has 12 input tokens, and each token embedding has dimension d_model = 512. What is the shape of the embedding matrix for this single sentence before batching?
- 3 Explain why positional encoding is necessary in a transformer and what kind of information would be lost without it.