The bias-variance tradeoff explains why a model that fits training data well may still predict new data poorly. Bias is error from using a model that is too simple, while variance is error from being too sensitive to the training data. The goal is not to make the most complex model possible, but to find the model complexity that gives the lowest prediction error.
This idea is central in statistics, machine learning, and scientific modeling because predictions must work on unseen data.
Key Facts
- Total error = Bias² + Variance + Irreducible error
- Bias measures systematic error caused by overly simple assumptions.
- Variance measures how much predictions change when the training data changes.
- As model complexity increases, bias usually decreases and variance usually increases.
- Underfitting occurs when bias is high and the model misses real patterns.
- Overfitting occurs when variance is high and the model learns noise instead of the signal.
Vocabulary
- Bias
- Bias is the error caused by a model making assumptions that are too simple to capture the true pattern.
- Variance
- Variance is the amount a model's predictions change when it is trained on different samples from the same population.
- Model complexity
- Model complexity describes how flexible a model is, such as the number of parameters, features, or curve bends it can use.
- Overfitting
- Overfitting happens when a model fits random noise in the training data and performs poorly on new data.
- Underfitting
- Underfitting happens when a model is too simple to capture the important structure in the data.
Common Mistakes to Avoid
- Choosing the model with the lowest training error. This is wrong because training error usually keeps decreasing as complexity increases, even when test error gets worse.
- Thinking bias and variance are always reduced together. This is wrong because increasing model flexibility often lowers bias but raises variance.
- Ignoring irreducible error. This is wrong because random noise and missing information can prevent perfect prediction even with the best possible model.
- Using the test set repeatedly to tune the model. This is wrong because the test set becomes part of the model selection process and no longer gives an honest estimate of future performance.
Practice Questions
- 1 A model has Bias² = 16, Variance = 9, and irreducible error = 4. What is the total expected prediction error?
- 2 Model A has Bias² = 25 and Variance = 4. Model B has Bias² = 9 and Variance = 18. If irreducible error is 3 for both, which model has lower total expected error?
- 3 A very flexible model has almost zero training error but much higher error on validation data. Explain whether this is more likely high bias or high variance, and describe one way to improve it.