Autoregressive decoding is the step by step process GPT style language models use to generate text. Instead of writing a whole sentence at once, the model predicts one token, adds it to the context, and then predicts the next token. This matters because each new choice depends on everything that came before it, which lets the model produce coherent paragraphs, code, and dialogue.
The method is simple in idea but powerful in practice because repeated next token prediction can build long and complex outputs.
Inside the model, the current token sequence is converted into vectors and processed by transformer layers that use self attention to combine information from earlier positions. The model outputs logits for every possible next token, then these scores are turned into probabilities with the softmax function. A decoding rule such as greedy choice, sampling, top k, or nucleus sampling selects the next token from that distribution.
The chosen token is appended to the sequence, and the loop repeats until a stop token is produced or a length limit is reached.
Understanding How GPT Models Generate Text
A token is not always a full word. It can be a common word, part of a long word, a space, punctuation, or a piece of code. This matters when judging a model's output.
A model may handle familiar word pieces well but struggle with rare names, unusual spellings, or languages that need many tokens per sentence. Token boundaries can produce odd errors, such as splitting a name in an unexpected place. They also affect cost and speed in real applications, since systems often count input and output by tokens rather than by words.
Logits are best understood as raw preference scores, not as probabilities or facts. A token with a larger logit is preferred relative to other available tokens at that moment. Softmax changes these relative scores into values that add up to one.
Small changes in logits can matter a great deal when several candidates have similar scores. Temperature controls how strongly the model favors its leading choices. A low temperature makes the distribution sharper, so the same common choices appear more often.
A high temperature spreads probability across more candidates. Higher temperature can help with brainstorming or varied creative writing, but it raises the chance of irrelevant details, invented claims, and broken formatting.
Sampling rules limit which candidates can be chosen before random selection occurs. Top k keeps only a fixed number of the most likely tokens. If top k is ten, every token outside the ten leading candidates is removed for that step.
Top p works differently. It keeps the smallest group of leading tokens whose combined probability reaches a chosen threshold. This makes the candidate group adapt to the situation.
When one answer is very likely, top p may keep only a few choices. When the model is uncertain, it may keep more.
Greedy decoding removes randomness by always selecting the leading token. It can be useful for repeatable tests, structured tasks, and debugging, though it may get trapped in dull wording or repetitive patterns.
Different runs can produce different text even with the same prompt because sampling uses random numbers to choose among permitted candidates. A fixed random seed can make a sampled run repeatable in a particular system, but changes to the model version, settings, prompt formatting, or available context can still change the result. Students should treat generated text as a plausible continuation, not as a checked source.
The model does not retrieve truth directly from its probability scores. It predicts patterns learned from data. Check quotations, calculations, dates, citations, and confident claims against reliable sources.
When experimenting, change one setting at a time and compare outputs. Notice how temperature, top k, top p, prompt wording, and stop conditions affect clarity, variety, repetition, and factual accuracy.
Key Facts
- Autoregressive factorization writes sequence probability as P(x1, x2, ..., xn) = product from t = 1 to n of P(xt | x1, ..., x(t-1)).
- At each step, the model computes logits z_i for each token in the vocabulary.
- Softmax converts logits to probabilities: P(token i) = e^(z_i) / sum over j of e^(z_j).
- Greedy decoding picks the highest probability token: x_next = argmax_i P(token i).
- Temperature rescales logits before softmax: P(token i) = e^(z_i/T) / sum over j of e^(z_j/T).
- The generation loop is context -> model -> probabilities -> token choice -> append token -> repeat.
Vocabulary
- Token
- A token is a basic unit of text used by the model, such as a word, subword, punctuation mark, or symbol.
- Logit
- A logit is an unnormalized score the model assigns to each possible next token before probabilities are computed.
- Softmax
- Softmax is a function that turns a list of logits into probabilities that add up to 1.
- Self attention
- Self attention is the mechanism that lets each position in the sequence use information from earlier tokens when forming its representation.
- Decoding
- Decoding is the procedure for choosing the next token from the model's predicted probability distribution.
Common Mistakes to Avoid
- Assuming the model generates an entire sentence in one pass, which is wrong because autoregressive decoding predicts one token at a time and updates the context after each choice.
- Treating logits as probabilities, which is wrong because logits can be any real numbers and must be passed through softmax to become a valid probability distribution.
- Thinking greedy decoding and sampling are the same, which is wrong because greedy always takes the highest probability token while sampling can choose lower probability tokens to increase variety.
- Ignoring the effect of temperature, which is wrong because changing T reshapes the probability distribution and can make output more deterministic or more random.
Practice Questions
- 1 A model assigns logits [2.0, 1.0, 0.0] to three possible next tokens A, B, and C. Using softmax, compute the probability of each token to three decimal places.
- 2 A model predicts next token probabilities: cat 0.50, dog 0.30, runs 0.20. Under greedy decoding, which token is chosen? If top k sampling uses k = 2, which tokens remain eligible for selection?
- 3 Explain why the sentence generated after 20 decoding steps can change if the model chooses a different token at step 3, even when all later model weights stay the same.