Gradient boosting is a machine learning method that combines many simple models to make one strong predictor. It is often used for tasks like predicting prices, sorting emails, detecting fraud, or estimating risk. The main idea is that each new model focuses on the mistakes made by the models before it.
This makes gradient boosting powerful, accurate, and useful in real data problems.
Key Facts
- Gradient boosting builds models one at a time, not all at once.
- Final prediction = first prediction + corrections from many weak learners.
- For regression, residual = actual value - predicted value.
- Each new tree tries to reduce the remaining error from earlier trees.
- Learning rate controls step size: new prediction = old prediction + learning rate × tree correction.
- Too many trees can cause overfitting, where the model memorizes training data instead of learning patterns.
Vocabulary
- Gradient boosting
- Gradient boosting is a machine learning method that combines many weak models, usually decision trees, to make a stronger model.
- Weak learner
- A weak learner is a simple model that performs only a little better than guessing but can become useful when combined with others.
- Decision tree
- A decision tree is a model that makes predictions by asking a sequence of yes or no questions about the data.
- Residual
- A residual is the difference between the actual value and the value predicted by the model.
- Learning rate
- The learning rate is a number that controls how strongly each new tree changes the model's prediction.
Common Mistakes to Avoid
- Thinking gradient boosting trains all trees independently is wrong because each tree is built after the previous ones and focuses on their errors.
- Using a learning rate that is too large is wrong because the model may jump too far when correcting errors and miss better predictions.
- Adding more trees without checking performance is wrong because the model can overfit and perform worse on new data.
- Confusing residuals with predictions is wrong because residuals are errors that show what the next tree should try to correct.
Practice Questions
- 1 A model first predicts 50 for a data point, but the actual value is 62. What is the residual?
- 2 A gradient boosting model currently predicts 40. A new tree suggests a correction of 8, and the learning rate is 0.25. What is the updated prediction?
- 3 Explain why gradient boosting often uses small decision trees instead of one very large decision tree.