Training a large language model happens in three distinct phases. Pretraining exposes the model to massive amounts of text and teaches it to predict the next token given what came before. The result is a base model that can complete text fluently but does not yet follow instructions or behave safely.
Fine-tuning then adapts the base model on curated examples of desired behavior, making it more useful for specific tasks. The final phase, Reinforcement Learning from Human Feedback (RLHF), uses human rankings to train a reward model, then uses that reward signal to further shape the LLM through reinforcement learning so its outputs align with human preferences.
Understanding these stages clarifies why models behave differently from each other. Two models with the same architecture can produce very different outputs depending on their pretraining data, fine-tuning examples, and whether RLHF was applied. It also explains common behaviors: base models are verbose and prone to continuing text in unexpected directions, while RLHF-tuned assistants tend to give concise, helpful answers and decline problematic requests.
Understanding How LLMs Are Trained
A model does not store sentences in the way a person remembers them. Text is first broken into tokens, which may be whole words, word pieces, spaces, or punctuation. Each token becomes a list of numbers called an embedding.
Inside a transformer, attention layers compare tokens across the available context. This helps the model connect a pronoun with an earlier noun, follow a pattern in code, or use facts mentioned several sentences earlier.
During training, it makes a prediction, measures how surprising the correct token was, then slightly changes billions of internal weights. Repeating this process over huge batches gradually builds useful patterns.
The quality of training data matters as much as the amount. Raw text can contain errors, duplicated pages, private information, unfair stereotypes, spam, and conflicting claims. Data pipelines try to filter some of this material, remove near duplicates, identify languages, and balance sources.
They cannot make a perfect dataset. A model can learn incorrect associations when they occur often in its data. It can produce a convincing answer without checking whether that answer is true.
This is one reason fluent writing is not proof of reliable knowledge. When using an LLM for schoolwork, students should verify important claims with textbooks, primary sources, or trusted reference sites.
Fine-tuning changes the model through a much smaller, more carefully selected dataset. The examples teach formats that users expect, such as explaining a solution step by step, writing a polite email, or refusing a request that could cause harm. Each example acts like a target behavior, but examples can conflict.
A model trained to be brief may leave out useful detail. One trained to be highly cautious may refuse harmless requests.
The choice of prompts, answers, languages, and cultural assumptions therefore affects the final assistant. Fine-tuning can improve behavior in a narrow area, yet it does not automatically give the model new dependable facts or perfect reasoning.
Human preference training has a further limitation. Raters usually choose between several generated answers under written guidelines. Their choices turn broad ideas like helpfulness, safety, and clarity into measurable signals.
The reward model is only an estimate of those judgments. If it rewards answers that sound confident, the language model may learn polished wording even when its reasoning is weak. Researchers watch for reward hacking, where a system finds ways to score well without doing the intended job.
Alternatives such as direct preference optimization use preference pairs more directly, while constitutional methods use written principles to guide critiques and revisions. Learning this pipeline helps students judge AI output as a designed product with strengths, blind spots, and human choices behind it.
Key Facts
- Pretraining: model learns to predict the next token from internet-scale text corpora.
- Objective: minimize cross-entropy loss - P(next token | all previous tokens).
- Base model: fluent but not instruction-following; useful for text completion only.
- Supervised fine-tuning (SFT): further training on curated prompt-response pairs.
- Reward model: trained on human comparisons (A is better than B) to score outputs.
- RLHF: uses the reward model signal to update the policy via PPO or similar RL algorithm.
- Constitutional AI and DPO are alternative alignment techniques to RLHF.
Vocabulary
- Pretraining
- The initial large-scale training phase where the model learns language structure from raw text by predicting the next token.
- Base model
- A model that has been pretrained but not yet fine-tuned or aligned; it completes text but does not follow instructions.
- Fine-tuning
- Additional training on a smaller, curated dataset to adapt the base model toward a specific behavior or task.
- RLHF
- Reinforcement Learning from Human Feedback - a technique that uses human preference rankings to train a reward model used to improve the LLM.
- Reward model
- A separate neural network trained on human comparisons that scores how good an LLM output is.
Common Mistakes to Avoid
- Thinking pretraining and fine-tuning are the same step - they are different phases with different objectives, dataset sizes, and compute requirements.
- Assuming a larger base model always gives better fine-tuned results - data quality and fine-tuning process matter at least as much as model size.
- Confusing RLHF with supervised learning - RLHF involves a reward signal and a policy update loop, not direct label supervision.
- Believing fine-tuned models have memorized all knowledge - factual knowledge mostly comes from pretraining, not fine-tuning.
Practice Questions
- 1 Explain in your own words what happens during pretraining and why the result is called a base model.
- 2 Why is a base model generally not used directly as an assistant, and what phases address this?
- 3 Describe how a reward model is trained and how it is used in RLHF.