Model distillation is a way to transfer knowledge from a large, powerful AI model into a smaller, faster one. The large model is called the teacher model, and the smaller model is called the student model. This matters because huge AI models can be expensive, slow, and hard to run on phones, laptops, or classroom computers.
Distillation helps make AI more practical while keeping much of its accuracy.
Key Facts
- Teacher model: a large trained model that produces helpful outputs for the student to learn from.
- Student model: a smaller model trained to imitate the teacher model while using less memory and computation.
- Distillation often uses soft labels, which are probability scores such as cat = 0.70, fox = 0.20, dog = 0.10.
- A common training goal is Loss = hard label loss + alpha × distillation loss.
- Compression ratio = teacher size / student size.
- Speedup = teacher inference time / student inference time.
Vocabulary
- Model distillation
- Model distillation is a training method where a smaller AI model learns to copy the behavior of a larger AI model.
- Teacher model
- A teacher model is a large, accurate model that provides outputs or probability scores for another model to learn from.
- Student model
- A student model is a smaller model trained to imitate the teacher while being faster or cheaper to run.
- Soft label
- A soft label is a set of probability values that shows how strongly a model thinks each answer might be correct.
- Inference
- Inference is the process of using a trained model to make a prediction on new input data.
Common Mistakes to Avoid
- Thinking distillation copies the teacher model exactly. This is wrong because the student is smaller and usually approximates the teacher rather than duplicating every detail.
- Training only on the teacher's top answer. This loses useful information because the full probability distribution shows which wrong answers are close and which are very unlikely.
- Assuming the student is always as accurate as the teacher. A smaller model can be faster and cheaper, but it may lose accuracy if it is too small or trained poorly.
- Ignoring the original correct labels. Distillation often works best when the student learns from both the teacher's outputs and the real labels from the dataset.
Practice Questions
- 1 A teacher model takes 200 milliseconds to answer, and a student model takes 40 milliseconds. What is the speedup using Speedup = teacher inference time / student inference time?
- 2 A teacher model uses 1,200 megabytes of memory, and a student model uses 150 megabytes. What is the compression ratio using Compression ratio = teacher size / student size?
- 3 A teacher model predicts probabilities for an image as cat = 0.60, fox = 0.25, and dog = 0.15. Explain why these soft labels can teach a student model more than the single hard label cat.