Temperature in AI text generation is a setting that controls how predictable or surprising an AI model's next word choice can be. It matters because the same prompt can produce a careful factual answer, a creative story, or a strange sentence depending on this setting. A low temperature makes the model favor the most likely words, while a high temperature gives less likely words a better chance.
This idea connects computer science with probability and statistics because the model is choosing from a distribution of possible next tokens.
Key Facts
- Temperature changes the shape of the probability distribution used to pick the next token.
- Low temperature, such as T = 0.2, makes high probability tokens dominate the choice.
- High temperature, such as T = 1.5, spreads probability more evenly across many tokens.
- A common formula is p_i = exp(z_i/T) / sum exp(z_j/T), where z_i is the model score for token i.
- T = 1 usually keeps the model's original probability distribution unchanged.
- Lower temperature improves consistency, while higher temperature can increase creativity and risk of errors.
Vocabulary
- Temperature
- Temperature is a setting that controls how random or predictable an AI model's token choices are.
- Token
- A token is a piece of text, such as a word, part of a word, number, or symbol, that an AI model processes.
- Probability distribution
- A probability distribution lists the possible outcomes and the chance that each one will be chosen.
- Logit
- A logit is a raw score a model gives to a possible next token before it is converted into a probability.
- Sampling
- Sampling is the process of randomly choosing one outcome according to its assigned probabilities.
Common Mistakes to Avoid
- Thinking temperature is a measure of intelligence, which is wrong because it changes randomness, not what the model knows.
- Setting temperature very high for factual answers, which is risky because it can make unlikely and incorrect words more likely.
- Assuming temperature guarantees creativity, which is wrong because it only changes probabilities and the model still depends on its training and prompt.
- Confusing the most likely token with the best answer, which is wrong because the most probable next word may be repetitive, generic, or not ideal for the task.
Practice Questions
- 1 An AI has next-token probabilities of cat = 0.70, dog = 0.20, and fox = 0.10 at T = 1. If a low temperature makes the highest probability token even more dominant, which token is most likely to be chosen and why?
- 2 A model generates 100 tokens using a setting where the word blue has probability 0.25 each time it appears as an option. About how many times would you expect blue to be chosen in 100 independent choices?
- 3 You are using an AI to write a safety instruction manual. Should you choose a low, medium, or high temperature, and what tradeoff are you making?