Machine learning models learn patterns from data so they can make predictions on new examples. Overfitting and underfitting describe two common ways a model can fail to learn the right pattern. An underfit model is too simple and misses important structure in the data.
An overfit model is too complicated and follows random noise instead of the true trend.
Key Facts
- Underfitting happens when a model is too simple to capture the real pattern in the data.
- Overfitting happens when a model learns noise or random details from the training data.
- A good fit balances simplicity and accuracy so it predicts new data well.
- Total error = bias error + variance error + irreducible error.
- High bias often causes underfitting, while high variance often causes overfitting.
- Test error is usually more important than training error because it measures performance on new data.
Vocabulary
- Training data
- Training data is the set of examples a machine learning model uses to learn a pattern.
- Test data
- Test data is a separate set of examples used to check how well a model works on new cases.
- Underfitting
- Underfitting occurs when a model is too simple and performs poorly on both training data and test data.
- Overfitting
- Overfitting occurs when a model matches the training data too closely and performs poorly on new data.
- Generalization
- Generalization is the ability of a model to make accurate predictions on data it has not seen before.
Common Mistakes to Avoid
- Choosing the model with the lowest training error only is wrong because an overfit model can memorize the training data and fail on new examples.
- Assuming a more complex model is always better is wrong because extra complexity can make the model chase noise instead of the real pattern.
- Using the same data for training and testing is wrong because it hides whether the model can generalize to new data.
- Calling every wrong prediction overfitting is wrong because errors can also come from underfitting, noisy data, missing features, or measurement mistakes.
Practice Questions
- 1 A model has training error 4% and test error 22%. Is this more likely overfitting or underfitting? Explain using the two error values.
- 2 Model A has training error 18% and test error 20%. Model B has training error 2% and test error 19%. Which model is more likely to generalize better, and why?
- 3 A scatterplot shows points forming a curved pattern. One model uses a straight line, one uses a smooth curve, and one wiggles through nearly every point. Explain which model is underfitting, which is a good fit, and which is overfitting.