Overfitting and underfitting describe two common ways a statistical or machine learning model can fail to generalize. An underfit model is too simple to capture the real pattern, while an overfit model is so complex that it learns noise in the training data. The goal is not to make the training error as small as possible, but to make accurate predictions on new data.
This makes the balance between bias and variance central to model building.
Key Facts
- Prediction error can be split conceptually into error = bias^2 + variance + irreducible error.
- Underfitting usually has high bias, meaning the model is too simple and misses important patterns.
- Overfitting usually has high variance, meaning the model changes too much when trained on different samples.
- As model complexity increases, training error usually decreases, but test error often decreases first and then increases.
- A good fit is often near the minimum of validation or test error, not necessarily the minimum of training error.
- Regularization adds a penalty such as total loss = training loss + λ × complexity penalty.
Vocabulary
- Overfitting
- Overfitting occurs when a model learns random noise or sample-specific details instead of the general pattern.
- Underfitting
- Underfitting occurs when a model is too simple to represent the main relationship in the data.
- Bias
- Bias is error caused by incorrect or overly simple assumptions in a model.
- Variance
- Variance is error caused by a model being too sensitive to the particular training data used.
- Regularization
- Regularization is a method that discourages overly complex models by adding a penalty to the loss function.
Common Mistakes to Avoid
- Choosing the model with the lowest training error is wrong because training error usually improves as complexity increases, even when the model is memorizing noise.
- Assuming more features always improve predictions is wrong because extra features can increase variance and make the model fit accidental patterns.
- Using the test set repeatedly during model selection is wrong because it makes the test set part of the training process and gives an overly optimistic estimate of performance.
- Interpreting regularization as deleting learning is wrong because regularization usually keeps the model flexible while reducing unnecessary complexity.
Practice Questions
- 1 A model has training error 0.04 and test error 0.25. Is this more likely overfitting or underfitting, and why?
- 2 Three models have validation errors 0.18, 0.11, and 0.16 at complexity levels 2, 5, and 12. Which complexity should be selected, and what does this suggest about the most complex model?
- 3 Explain why the best model in the bias-variance tradeoff is usually not the simplest model and not the most complex model.