Transformers process all tokens in a sequence at the same time, which makes them fast and powerful for language, vision, and many other tasks. But this parallel processing creates a problem: the model does not automatically know the order of the tokens. Positional encoding solves this by adding information about each token's position in the sequence.
Without it, a transformer would struggle to tell the difference between sentences with the same words in different orders.
In practice, each token first becomes a vector called an embedding, and then a positional vector is combined with it before the data enters the attention layers. The combined representation lets attention use both meaning and location when comparing tokens. Some transformers use fixed sinusoidal encodings, while others learn position vectors during training.
More advanced models may use relative position methods so the model focuses on distances between tokens instead of only absolute index values.
Understanding Positional Encoding Explained
Attention works by comparing one token representation with representations from other tokens. Each representation is turned into a query, a key, and a value. A query dot key score measures how strongly one token should attend to another.
If position information were removed, swapping two input tokens would merely swap rows of these calculations. The attention pattern would contain the same relationships, just in a different row order. This matters because grammar depends on direction and distance.
In the phrase "dog bites man," the first noun is usually the actor. In "man bites dog," the roles change. Position features give the query and key projections clues that help the model learn such patterns.
Fixed sine and cosine encodings use many waves with different wavelengths. Some vector dimensions change rapidly from one position to the next. Other dimensions change slowly across hundreds or thousands of positions.
Together, these waves create a distinct location pattern. Their useful property is that a shift by a certain number of tokens changes the pattern in a predictable way. A model can therefore learn to recognize nearby tokens or tokens separated by a familiar distance.
Fixed waves do not need training, so they can provide positions that were not present in the training set. Still, performance at much longer lengths is not guaranteed. The model must have learned how to use the unfamiliar combinations of wave values.
Learned position embeddings store a separate trainable vector for each allowed location. They can fit the typical sentence lengths and layout patterns in a training set very well. Their weakness is a hard limit or a weak guess beyond the largest trained position.
Relative methods handle a different idea. They describe how far apart two tokens are, such as one token before, two tokens after, or far away. Rotary position encoding, often called RoPE, applies a position-dependent rotation to paired components of queries and keys.
The angle of rotation changes with location. When a rotated query is compared with a rotated key, the score naturally contains information about their separation. This is useful for long contexts, where the distance between a pronoun and the noun it refers to can matter more than either absolute location.
Position information is not the same as a rule about which tokens may be read. A language model that predicts the next token usually uses a causal mask. The mask blocks attention from future tokens during training and generation.
Positional encoding then tells the model where the visible earlier tokens are. Students often confuse these two jobs. Another useful check is to trace a short sentence through the model.
Identify token positions, then consider which words should attend to each other for agreement, negation, or reference. In code, pay attention to indexing conventions, padding tokens, and maximum sequence length. A shifted index or incorrect padding mask can give a model misleading location signals even when the attention code itself is correct.
Key Facts
- Transformers need position information because self attention alone is permutation invariant.
- Input to the model is often x_i = e_i + p_i, where e_i is the token embedding and p_i is the positional encoding for position i.
- A common fixed encoding uses PE(pos, 2i) = sin(pos / 10000^(2i/d_model)).
- The paired odd dimension is PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)).
- In self attention, scores are computed by Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V.
- Relative position methods encode token distance, which can help generalization to longer sequences.
Vocabulary
- Token embedding
- A token embedding is a vector that represents the meaning or identity of a token in a continuous numerical space.
- Positional encoding
- Positional encoding is a vector added to or combined with a token embedding so the model knows where the token appears in the sequence.
- Self attention
- Self attention is a mechanism that lets each token compare itself with other tokens in the same sequence to gather useful context.
- Absolute position
- Absolute position is the exact index of a token in the sequence, such as first, second, or third.
- Relative position
- Relative position describes how far apart two tokens are, such as one step away or three steps away.
Common Mistakes to Avoid
- Assuming token embeddings already contain word order, which is wrong because embeddings mainly represent token identity or meaning and not sequence position by themselves.
- Treating positional encoding as optional in a basic transformer, which is wrong because without position information the model cannot reliably distinguish reordered sequences.
- Thinking sinusoidal encodings are random patterns, which is wrong because the sine and cosine functions create structured position signals that vary smoothly across dimensions.
- Confusing absolute and relative position methods, which is wrong because absolute methods encode each token's index while relative methods encode distances between tokens.
Practice Questions
- 1 A transformer uses d_model = 8. For position pos = 0, compute PE(0, 0) and PE(0, 1) using PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) and PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)).
- 2 A token embedding is e = [1, 0, 2, -1] and its positional encoding is p = [0.1, 0.2, -0.1, 0.3]. Find the combined input vector x = e + p.
- 3 Explain why a transformer without positional encoding would have trouble telling the difference between the sequences ["dog bites man"] and ["man bites dog"] even if it sees the same three tokens.