Reinforcement Learning from Human Feedback, or RLHF, is a way to train AI systems using people’s judgments about which answers are better. It matters because many useful tasks, such as writing a helpful explanation or giving safe advice, do not have one simple right answer. Human feedback helps the AI learn qualities like clarity, accuracy, helpfulness, and politeness.
This makes RLHF important for chatbots, tutoring tools, coding assistants, and other systems that interact with people.
Key Facts
- RLHF loop: AI gives answers → humans rank answers → rankings become data → model learns preferences → AI improves.
- A reward model predicts which AI response a human would prefer.
- Training data often looks like pairs of responses, such as Response A is preferred over Response B.
- A policy is the model’s strategy for choosing the next word or action.
- The goal is to maximize expected reward: choose outputs that receive higher preference scores.
- RLHF can improve helpfulness and safety, but it can also reflect bias or mistakes in the human feedback.
Vocabulary
- Reinforcement Learning from Human Feedback
- A training method where an AI model improves by using human preferences as a learning signal.
- Reward Model
- A model trained to score AI responses based on which ones humans are likely to prefer.
- Preference Data
- Data made from human comparisons, rankings, or ratings of different AI outputs.
- Policy
- The rule or strategy an AI model uses to choose its next response or action.
- Training Loop
- A repeated cycle where a model makes outputs, receives feedback, updates, and tries again.
Common Mistakes to Avoid
- Thinking RLHF means humans write every answer, which is wrong because humans usually rank or rate sample answers so the model can learn patterns from feedback.
- Assuming the highest ranked answer is always factually true, which is wrong because humans can prefer an answer that sounds good but contains an error.
- Ignoring the reward model, which is wrong because RLHF usually trains a separate model to predict human preferences before improving the main AI model.
- Treating RLHF as a one-time step, which is wrong because useful systems often need repeated feedback, testing, and updates to improve safely.
Practice Questions
- 1 An AI gives 4 possible answers to a question. A human ranks them from best to worst. How many adjacent pair comparisons are there if you compare 1st vs 2nd, 2nd vs 3rd, and 3rd vs 4th?
- 2 A reward model gives three responses scores of 0.25, 0.70, and 0.55. Which response would the AI be encouraged to choose if it only maximizes reward, and by how much is it higher than the second-best score?
- 3 A student says RLHF makes an AI perfectly safe because humans check its answers. Explain why this claim is too strong and name one limitation of human feedback.