Machine learning models learn patterns from data so they can make predictions on new examples. The bias-variance tradeoff explains why a model can fail by being too simple or too sensitive to small details in the training data. A good model is not the one that memorizes perfectly, but the one that makes accurate predictions on data it has never seen before.
This idea matters in AI because it helps engineers choose models that are useful, fair, and reliable.
Key Facts
- Prediction error can be thought of as Error = Bias^2 + Variance + Irreducible error.
- High bias means the model is too simple and underfits the data.
- High variance means the model is too complex and overfits the training data.
- Training error measures performance on examples the model learned from.
- Test error measures performance on new examples and is usually more important than training error.
- A good model balances bias and variance by keeping test error low, not just training error low.
Vocabulary
- Bias
- Bias is the error caused when a model makes overly simple assumptions about the data.
- Variance
- Variance is the error caused when a model changes too much in response to small changes in the training data.
- Underfitting
- Underfitting happens when a model is too simple to capture the real pattern in the data.
- Overfitting
- Overfitting happens when a model memorizes training data details instead of learning a pattern that works on new data.
- Generalization
- Generalization is a model's ability to make accurate predictions on new examples it did not train on.
Common Mistakes to Avoid
- Choosing the model with the lowest training error only. This is wrong because a model can memorize the training data and still perform poorly on new data.
- Assuming a more complex model is always better. This is wrong because extra complexity can increase variance and cause overfitting.
- Ignoring test or validation data. This is wrong because you need unseen data to estimate how well the model generalizes.
- Thinking bias and variance are separate problems that never interact. This is wrong because reducing one often increases the other, so model design requires a balance.
Practice Questions
- 1 A model has Bias^2 = 9, Variance = 4, and Irreducible error = 2. Using Error = Bias^2 + Variance + Irreducible error, what is the total expected error?
- 2 Model A has training error 2% and test error 18%. Model B has training error 8% and test error 10%. Which model generalizes better, and by how many percentage points is its test error lower?
- 3 A student trains a model that performs badly on both the training set and the test set. Explain whether this is more likely high bias or high variance, and describe one way to improve the model.