Regularization is a method used in machine learning to help a model learn patterns without memorizing random noise. This matters because real data is often messy, with outliers, measurement errors, and accidental patterns. A regularized model usually makes better predictions on new examples because it stays simpler and more stable.
In an infographic, you can picture it as a smart robot brain choosing a smooth curve through noisy data points instead of a wild zigzag curve.
Key Facts
- Overfitting happens when a model fits training data too closely but performs poorly on new data.
- Regularization adds a penalty for unnecessary complexity to the loss function.
- Regularized loss = original loss + penalty term.
- L2 regularization often uses loss = error + lambda sum(w^2), where w values are model weights.
- L1 regularization often uses loss = error + lambda sum(|w|), which can shrink some weights to zero.
- A larger lambda usually means a simpler model, but too much regularization can cause underfitting.
Vocabulary
- Regularization
- Regularization is a technique that discourages a machine learning model from becoming too complex.
- Overfitting
- Overfitting occurs when a model learns noise or random details in the training data instead of the main pattern.
- Loss Function
- A loss function is a formula that measures how wrong a model's predictions are.
- Lambda
- Lambda is a tuning value that controls how strongly regularization penalizes complexity.
- Model Weight
- A model weight is a number inside a machine learning model that controls how much an input affects the prediction.
Common Mistakes to Avoid
- Thinking regularization makes the model more accurate on the training data. It often makes training error slightly higher so the model can perform better on new data.
- Choosing lambda without testing. Lambda must be tuned because a value that is too small may not reduce overfitting, while a value that is too large may erase useful patterns.
- Confusing overfitting with underfitting. Overfitting means the model is too complex for the data, while underfitting means the model is too simple to capture the real pattern.
- Assuming all regularization methods work the same way. L1 can remove weak features by shrinking weights to zero, while L2 usually keeps features but makes weights smaller.
Practice Questions
- 1 A model has original loss 8.0, lambda = 0.1, and weights 3, 4, and 0. Using L2 regularization, calculate the regularized loss with loss = error + lambda sum(w^2).
- 2 A model has original loss 12.0, lambda = 0.5, and weights -2, 1, and 3. Using L1 regularization, calculate the regularized loss with loss = error + lambda sum(|w|).
- 3 A model draws a very wavy curve that passes through almost every noisy training point, but it performs poorly on new data. Explain whether regularization should be increased or decreased, and why.