Gradient boosting is a machine learning method that combines many simple models to make one strong predictor. It is often used for tasks like predicting prices, sorting emails, detecting fraud, or estimating risk. The main idea is that each new model focuses on the mistakes made by the models before it.
This makes gradient boosting powerful, accurate, and useful in real data problems.
Understanding AI & Machine Learning: Gradient Boosting Explained
The small models used in gradient boosting are usually decision trees with only a few levels. A tree divides data into groups by applying simple conditions. For house prices, one branch might separate homes by area, while another separates them by location.
At the end of each branch is a number that represents a correction. A shallow tree cannot understand every detail of the data.
That limitation is useful because it forces each tree to find one small pattern rather than inventing a complicated rule. Over many stages, these small adjustments can describe curved, uneven relationships that a single simple formula would miss.
The word gradient comes from optimisation, which is the process of reducing a measure of error called a loss. A gradient gives the direction in which the loss falls fastest. For a price prediction, the remaining difference between a real price and a predicted price gives a useful signal for the next correction.
Classification needs a different signal because the output is often a probability. A spam filter, for example, must be penalised more when it confidently labels spam as safe than when it is uncertain.
Gradient boosting uses the loss function to calculate suitable correction targets. This is why the method can work with several kinds of prediction problems.
The size and shape of the trees matter as much as the number of trees. Deep trees can capture detailed interactions, such as a discount mattering only for customers in one region. They can easily fit random noise as well.
A small learning rate makes every correction cautious. This usually means that more trees are needed, though the result can be more stable. Developers often keep aside a validation set that is not used to fit the trees.
They watch its error while training and stop when that error no longer improves. Some versions use a random sample of rows or features for each tree. This adds variation and can reduce reliance on accidental patterns.
Students meet these ideas whenever an app ranks search results, estimates delivery times, flags unusual card activity, or predicts demand for a shop. The quality of the input data sets a hard limit on the model. Missing values, wrongly recorded labels, and data collected after the event being predicted can produce misleading results.
Using future information to predict the past is called data leakage. Time based data needs a time based train and test split for this reason. Feature importance scores can help show which inputs influenced the model, but they do not prove cause and effect.
A model may use a postcode because it is linked to income or access to services. That can create unfair outcomes, so accuracy should be checked across different groups, not only as one overall score.
Key Facts
- Gradient boosting builds models one at a time, not all at once.
- Final prediction = first prediction + corrections from many weak learners.
- For regression, residual = actual value - predicted value.
- Each new tree tries to reduce the remaining error from earlier trees.
- Learning rate controls step size: new prediction = old prediction + learning rate × tree correction.
- Too many trees can cause overfitting, where the model memorizes training data instead of learning patterns.
Vocabulary
- Gradient boosting
- Gradient boosting is a machine learning method that combines many weak models, usually decision trees, to make a stronger model.
- Weak learner
- A weak learner is a simple model that performs only a little better than guessing but can become useful when combined with others.
- Decision tree
- A decision tree is a model that makes predictions by asking a sequence of yes or no questions about the data.
- Residual
- A residual is the difference between the actual value and the value predicted by the model.
- Learning rate
- The learning rate is a number that controls how strongly each new tree changes the model's prediction.
Common Mistakes to Avoid
- Thinking gradient boosting trains all trees independently is wrong because each tree is built after the previous ones and focuses on their errors.
- Using a learning rate that is too large is wrong because the model may jump too far when correcting errors and miss better predictions.
- Adding more trees without checking performance is wrong because the model can overfit and perform worse on new data.
- Confusing residuals with predictions is wrong because residuals are errors that show what the next tree should try to correct.
Practice Questions
- 1 A model first predicts 50 for a data point, but the actual value is 62. What is the residual?
- 2 A gradient boosting model currently predicts 40. A new tree suggests a correction of 8, and the learning rate is 0.25. What is the updated prediction?
- 3 Explain why gradient boosting often uses small decision trees instead of one very large decision tree.