Gradient descent is a method that helps computers learn by improving a model step by step. In machine learning, a model makes predictions, measures how wrong they are, and then adjusts itself to reduce that error. The error is often called the loss, and the goal is to make the loss as small as possible.
This idea matters because it powers many AI systems, from image recognition to recommendation tools.
Understanding AI & Machine Learning: Gradient Descent Explained
Imagine a model that predicts house prices from floor area. It may begin with a random rule, such as giving too much importance to each extra square metre. Training data reveals a pattern in its mistakes.
If predictions tend to be too high when the model weight increases, the slope of the loss with respect to that weight is positive. The training process then lowers the weight a little. If raising the weight would reduce the mistakes, the slope is negative, so the process raises the weight.
This happens separately for every adjustable number in the model. Those numbers are called parameters or weights.
The slope is not found by trying every possible setting one at a time. Modern models calculate it efficiently with a process called backpropagation. A prediction travels forward through layers of calculations.
The loss compares that prediction with the correct training answer. Backpropagation then works backward through the same calculations. It determines how much each weight contributed to the final error.
This is especially important in neural networks, which may contain millions of weights. Without this backward calculation, training would take far too long to be practical.
A model usually learns from a small batch of examples at each step, rather than the whole training set. This makes each update much faster and lets the model train on very large data sets. The batch gives only an estimate of the true best direction, so the loss may bounce up and down during training.
That noise is normal. Across many steps, the general trend should improve.
Students often see a graph of training loss fall quickly at first, then flatten. This means the model is finding smaller improvements as it gets closer to a useful setting.
The lowest training loss is not the only goal. A model can memorise its training examples, then perform badly on new examples. This is called overfitting.
To check for it, developers keep some data aside as validation data. They monitor its loss while training. If training loss keeps falling but validation loss starts rising, training may need to stop.
Learning rate choice matters here, but so do data quality, model size, and the number of training steps. When learning this topic, pay attention to the difference between a parameter, a gradient, a batch, and a loss. They have different jobs, yet they work together during every update.
Key Facts
- Loss measures prediction error, so smaller loss usually means a better model.
- Gradient descent update rule: new value = old value - learning rate × gradient.
- For one parameter w: w_new = w_old - α dL/dw.
- The gradient points in the direction where the loss increases fastest.
- Gradient descent moves opposite the gradient to go downhill on the loss landscape.
- A learning rate that is too large can overshoot the minimum, while one that is too small can learn very slowly.
Vocabulary
- Gradient descent
- Gradient descent is an optimization method that repeatedly adjusts a model's parameters to reduce its loss.
- Loss function
- A loss function is a formula that measures how far a model's predictions are from the correct answers.
- Gradient
- A gradient is a set of slopes that tells how the loss changes when each parameter changes.
- Learning rate
- The learning rate is a number that controls how big each update step is during training.
- Minimum
- A minimum is a low point of the loss function where the model has less error than nearby parameter values.
Common Mistakes to Avoid
- Thinking gradient descent always finds the absolute best answer. This is wrong because complex loss landscapes can have local minima, flat regions, or other challenges.
- Using a very large learning rate to make training faster. This is wrong because the updates may overshoot the minimum and make the loss bounce around or increase.
- Using a very small learning rate without checking progress. This is wrong because the model may improve so slowly that training takes too long.
- Moving in the same direction as the gradient. This is wrong because the gradient points uphill, while gradient descent must move in the opposite direction to reduce loss.
Practice Questions
- 1 A model has one parameter w = 6, learning rate α = 0.2, and gradient dL/dw = 4. Use w_new = w_old - α dL/dw to find the new value of w.
- 2 A model's loss values after five training steps are 20, 14, 9, 7, and 6. What is the total decrease in loss from the first step to the fifth step?
- 3 A student increases the learning rate and notices that the loss starts jumping up and down instead of steadily decreasing. Explain what is likely happening and how the student could fix it.