Machine learning models learn patterns from data so they can make predictions on new examples. Overfitting and underfitting describe two common ways a model can fail to learn the right pattern. An underfit model is too simple and misses important structure in the data.
An overfit model is too complicated and follows random noise instead of the true trend.
Understanding AI & Machine Learning: Overfitting vs Underfitting
A model is built by adjusting internal settings, often called parameters, until its predictions match examples it has already seen. This process uses a training set. However, matching old examples is not the final goal.
The important test is whether the same learned rule works on unfamiliar examples. Schools use this idea when students practise with one set of problems, then sit an exam with different questions.
Memorising every practice answer may produce a good practice score without building understanding. A useful model needs to learn the underlying relationship, not just store details from its training set.
Data is usually divided into separate groups for training, validation, and testing. The training group changes the model. The validation group helps people choose settings, such as the number of layers in a neural network, the depth of a decision tree, or the amount of regularisation.
Regularisation is a method that discourages a model from relying too heavily on tiny details. The test group is kept aside until the end.
Looking at test results repeatedly while making choices can quietly turn the test set into part of training. That gives an unrealistically positive result.
Learning curves can reveal what is going wrong. A learning curve compares error on training data with error on validation data as more examples are used. When both errors are large and close together, the model may lack enough flexibility or useful input features.
When training error is very small but validation error stays much larger, the model may be fitting accidental features of the sample. More training data often helps in this second case because random details are less likely to appear consistently. Adding more data does not always fix a model that is fundamentally too simple.
Bias and variance describe two different sources of avoidable mistakes. Bias comes from assumptions that are too restrictive. For example, fitting a straight line to a curved relationship creates a consistent pattern of error.
Variance comes from a model changing too much when its training sample changes. A very detailed decision tree can produce different rules from two similar data sets.
Some error remains even with an excellent model because measurements can be noisy, people can behave unpredictably, or important information may be missing. This unavoidable part is called irreducible error.
Students meet these ideas in systems that suggest videos, filter spam, recognise speech, estimate travel time, or help doctors examine scans. A model can appear accurate overall while performing poorly for a smaller group that was not well represented in its data. When judging a model, check which data was used, whether the test examples were truly separate, and which kinds of errors matter most.
In medical screening, missing a disease may be more serious than a false warning. In spam filtering, blocking an important message may be worse than allowing one unwanted email through. Good evaluation measures reflect those real consequences.
Key Facts
- Underfitting happens when a model is too simple to capture the real pattern in the data.
- Overfitting happens when a model learns noise or random details from the training data.
- A good fit balances simplicity and accuracy so it predicts new data well.
- Total error = bias error + variance error + irreducible error.
- High bias often causes underfitting, while high variance often causes overfitting.
- Test error is usually more important than training error because it measures performance on new data.
Vocabulary
- Training data
- Training data is the set of examples a machine learning model uses to learn a pattern.
- Test data
- Test data is a separate set of examples used to check how well a model works on new cases.
- Underfitting
- Underfitting occurs when a model is too simple and performs poorly on both training data and test data.
- Overfitting
- Overfitting occurs when a model matches the training data too closely and performs poorly on new data.
- Generalization
- Generalization is the ability of a model to make accurate predictions on data it has not seen before.
Common Mistakes to Avoid
- Choosing the model with the lowest training error only is wrong because an overfit model can memorize the training data and fail on new examples.
- Assuming a more complex model is always better is wrong because extra complexity can make the model chase noise instead of the real pattern.
- Using the same data for training and testing is wrong because it hides whether the model can generalize to new data.
- Calling every wrong prediction overfitting is wrong because errors can also come from underfitting, noisy data, missing features, or measurement mistakes.
Practice Questions
- 1 A model has training error 4% and test error 22%. Is this more likely overfitting or underfitting? Explain using the two error values.
- 2 Model A has training error 18% and test error 20%. Model B has training error 2% and test error 19%. Which model is more likely to generalize better, and why?
- 3 A scatterplot shows points forming a curved pattern. One model uses a straight line, one uses a smooth curve, and one wiggles through nearly every point. Explain which model is underfitting, which is a good fit, and which is overfitting.