Large language models choose words by assigning a probability to many possible next tokens, then selecting one according to a sampling rule. Top-k and top-p sampling are two common ways to control that choice so the output is not too random or too repetitive. They matter because small changes in sampling can make an AI response feel more creative, more focused, or more predictable.
Understanding these methods connects computer science with probability and statistics.
Key Facts
- A language model predicts P(next token | previous tokens), the probability of each possible next token.
- Top-k sampling keeps only the k highest-probability tokens before sampling from them.
- Top-p sampling keeps the smallest set of tokens whose total probability is at least p.
- After filtering, probabilities are renormalized so the kept tokens add to 1.
- If kept probabilities are 0.40, 0.30, and 0.10, the renormalized probabilities are 0.50, 0.375, and 0.125.
- Lower k or lower p usually makes output more focused, while higher k or higher p usually makes output more varied.
Vocabulary
- Token
- A token is a piece of text, such as a word, part of a word, number, or punctuation mark, that an AI model processes.
- Probability distribution
- A probability distribution lists possible outcomes and the probability of each outcome, with all probabilities adding to 1.
- Top-k sampling
- Top-k sampling is a method that keeps only the k most likely next tokens and randomly chooses from that smaller group.
- Top-p sampling
- Top-p sampling, also called nucleus sampling, keeps the most likely tokens until their total probability reaches a chosen cutoff p.
- Renormalization
- Renormalization is the process of rescaling the remaining probabilities so they add up to 1 after some options are removed.
Common Mistakes to Avoid
- Thinking top-k always chooses the single most likely word. This is wrong because top-k filters the list, then still samples randomly from the remaining k tokens.
- Forgetting to renormalize after filtering. This is wrong because the remaining probabilities no longer add to 1 until they are rescaled.
- Assuming top-p always keeps the same number of tokens. This is wrong because the number kept depends on the shape of the probability distribution.
- Using very low k or p and expecting creative writing. This is wrong because strict filtering removes many surprising or unusual word choices.
Practice Questions
- 1 A model assigns next-token probabilities: cat = 0.35, dog = 0.25, bird = 0.15, fish = 0.10, car = 0.08, moon = 0.07. With top-k sampling and k = 3, which tokens are kept, and what are their renormalized probabilities?
- 2 A model assigns probabilities in order: A = 0.42, B = 0.21, C = 0.14, D = 0.10, E = 0.08, F = 0.05. With top-p sampling and p = 0.80, which tokens are kept, and what is the total probability before renormalization?
- 3 A chatbot is giving dull, repetitive answers. Explain whether increasing k, increasing p, or both could help, and describe one possible risk of making the values too high.