Overfitting and underfitting describe two common ways a statistical or machine learning model can fail to generalize. An underfit model is too simple to capture the real pattern, while an overfit model is so complex that it learns noise in the training data. The goal is not to make the training error as small as possible, but to make accurate predictions on new data.
This makes the balance between bias and variance central to model building.
Understanding Statistics: Overfitting and Underfitting
A model learns by looking for a rule that connects inputs to an outcome. The inputs might include hours studied, attendance, and past scores. The outcome could be an exam result.
If the rule is very restricted, it may force every student into nearly the same trend. Real data often contains curves, thresholds, groups, or interactions between variables. For example, extra study time may help greatly at first, then help less after many hours.
A straight line can miss this shape. This is why choosing the model form matters before any computer produces a prediction.
Very flexible models have a different problem. A dataset is never perfectly clean. A test score can be affected by sleep, illness, an unclear question, or a marking mistake.
These accidental effects are not useful patterns for future students. A flexible model can create many bends or detailed rules to match them anyway. It may treat one unusual student as evidence for a general rule.
The model then gives a very accurate account of the data it saw, yet performs poorly on the next group. More data can reduce this risk because one random event has less influence when many observations are available.
Students can spot this issue by separating data into jobs. Training data is used to build the model. Validation data is used to compare choices, such as the number of features, the depth of a decision tree, or the strength of a penalty.
Test data is kept aside until the final evaluation. It gives a more honest estimate because the model has not been adjusted to please it. If the same validation set is checked repeatedly while making many choices, it can gradually become part of the design process.
This can make its result too optimistic. Cross validation helps when data is limited. It repeats the process with different sections held out, then averages the results.
Regularization is one practical way to control unnecessary complexity. It makes complicated models pay a cost for using very large coefficients or too many detailed splits. The exact rule depends on the method, but the purpose is the same.
Keep patterns that appear reliably and reduce patterns that only appear in one sample. Feature selection can help for the same reason. Adding every available measurement is not automatically sensible.
Some measurements are irrelevant, duplicated, or collected after the prediction would need to be made. When studying model results, compare training and validation errors, inspect unusual data points, and think about how the model will be used in the real world. A useful model should work on future cases collected under conditions similar to those expected in practice.
Key Facts
- Prediction error can be split conceptually into error = bias^2 + variance + irreducible error.
- Underfitting usually has high bias, meaning the model is too simple and misses important patterns.
- Overfitting usually has high variance, meaning the model changes too much when trained on different samples.
- As model complexity increases, training error usually decreases, but test error often decreases first and then increases.
- A good fit is often near the minimum of validation or test error, not necessarily the minimum of training error.
- Regularization adds a penalty such as total loss = training loss + λ × complexity penalty.
Vocabulary
- Overfitting
- Overfitting occurs when a model learns random noise or sample-specific details instead of the general pattern.
- Underfitting
- Underfitting occurs when a model is too simple to represent the main relationship in the data.
- Bias
- Bias is error caused by incorrect or overly simple assumptions in a model.
- Variance
- Variance is error caused by a model being too sensitive to the particular training data used.
- Regularization
- Regularization is a method that discourages overly complex models by adding a penalty to the loss function.
Common Mistakes to Avoid
- Choosing the model with the lowest training error is wrong because training error usually improves as complexity increases, even when the model is memorizing noise.
- Assuming more features always improve predictions is wrong because extra features can increase variance and make the model fit accidental patterns.
- Using the test set repeatedly during model selection is wrong because it makes the test set part of the training process and gives an overly optimistic estimate of performance.
- Interpreting regularization as deleting learning is wrong because regularization usually keeps the model flexible while reducing unnecessary complexity.
Practice Questions
- 1 A model has training error 0.04 and test error 0.25. Is this more likely overfitting or underfitting, and why?
- 2 Three models have validation errors 0.18, 0.11, and 0.16 at complexity levels 2, 5, and 12. Which complexity should be selected, and what does this suggest about the most complex model?
- 3 Explain why the best model in the bias-variance tradeoff is usually not the simplest model and not the most complex model.