The bias-variance tradeoff explains why a model that fits training data well may still predict new data poorly. Bias is error from using a model that is too simple, while variance is error from being too sensitive to the training data. The goal is not to make the most complex model possible, but to find the model complexity that gives the lowest prediction error.
This idea is central in statistics, machine learning, and scientific modeling because predictions must work on unseen data.
Understanding Statistics: The Bias-Variance Tradeoff
The tradeoff becomes clearer if you imagine building the same model many times. Each time, you collect a different sample from the same population. A stable model gives similar predictions across these samples.
An unstable model can produce very different predictions because a few selected cases have too much influence. This matters most when the training set is small, noisy, or unrepresentative.
A single unusual observation may pull a flexible curve toward it. The curve then describes an accident of the sample rather than a dependable feature of the population.
Consider a model for house prices. A very simple model might use only house size and draw one straight trend. It may miss important price changes linked to neighbourhood, condition, or distance from transport.
A highly flexible model might bend sharply to match every sale in the records, including a bargain sale between relatives or a house with unusual damage. Those details are real in the data, but they may not help estimate the price of the next house. Model choices such as the number of features, the depth of a decision tree, or the degree of a fitted curve control how much freedom the model has.
Scientists estimate useful model complexity by holding back some data from training. The held back data acts like a small rehearsal for future use. Cross validation improves this process by making several different training and checking splits.
It gives a less fragile estimate than trusting one split alone. A final test set should stay untouched until the model choices are finished.
If people repeatedly adjust a model after viewing test results, the test set quietly becomes part of training. This can make reported performance look better than it will be in real use.
Some prediction error cannot be removed by better modeling. Measurements can be imperfect, important causes may be missing, and parts of the future are genuinely unpredictable. Weather forecasts, exam score predictions, medical risk estimates, and recommendations all face this limit.
Students should learn to inspect both training performance and validation performance. Low scores on both suggest that the model has not captured enough structure.
A large gap, with excellent training results but weaker validation results, signals that the model may be relying too heavily on sample details. More data, simpler features, regularization, pruning, and averaging several models can often make predictions more reliable.
Key Facts
- Total error = Bias² + Variance + Irreducible error
- Bias measures systematic error caused by overly simple assumptions.
- Variance measures how much predictions change when the training data changes.
- As model complexity increases, bias usually decreases and variance usually increases.
- Underfitting occurs when bias is high and the model misses real patterns.
- Overfitting occurs when variance is high and the model learns noise instead of the signal.
Vocabulary
- Bias
- Bias is the error caused by a model making assumptions that are too simple to capture the true pattern.
- Variance
- Variance is the amount a model's predictions change when it is trained on different samples from the same population.
- Model complexity
- Model complexity describes how flexible a model is, such as the number of parameters, features, or curve bends it can use.
- Overfitting
- Overfitting happens when a model fits random noise in the training data and performs poorly on new data.
- Underfitting
- Underfitting happens when a model is too simple to capture the important structure in the data.
Common Mistakes to Avoid
- Choosing the model with the lowest training error. This is wrong because training error usually keeps decreasing as complexity increases, even when test error gets worse.
- Thinking bias and variance are always reduced together. This is wrong because increasing model flexibility often lowers bias but raises variance.
- Ignoring irreducible error. This is wrong because random noise and missing information can prevent perfect prediction even with the best possible model.
- Using the test set repeatedly to tune the model. This is wrong because the test set becomes part of the model selection process and no longer gives an honest estimate of future performance.
Practice Questions
- 1 A model has Bias² = 16, Variance = 9, and irreducible error = 4. What is the total expected prediction error?
- 2 Model A has Bias² = 25 and Variance = 4. Model B has Bias² = 9 and Variance = 18. If irreducible error is 3 for both, which model has lower total expected error?
- 3 A very flexible model has almost zero training error but much higher error on validation data. Explain whether this is more likely high bias or high variance, and describe one way to improve it.