Gradient descent is a method that helps computers learn by improving a model step by step. In machine learning, a model makes predictions, measures how wrong they are, and then adjusts itself to reduce that error. The error is often called the loss, and the goal is to make the loss as small as possible.
This idea matters because it powers many AI systems, from image recognition to recommendation tools.
Key Facts
- Loss measures prediction error, so smaller loss usually means a better model.
- Gradient descent update rule: new value = old value - learning rate × gradient.
- For one parameter w: w_new = w_old - α dL/dw.
- The gradient points in the direction where the loss increases fastest.
- Gradient descent moves opposite the gradient to go downhill on the loss landscape.
- A learning rate that is too large can overshoot the minimum, while one that is too small can learn very slowly.
Vocabulary
- Gradient descent
- Gradient descent is an optimization method that repeatedly adjusts a model's parameters to reduce its loss.
- Loss function
- A loss function is a formula that measures how far a model's predictions are from the correct answers.
- Gradient
- A gradient is a set of slopes that tells how the loss changes when each parameter changes.
- Learning rate
- The learning rate is a number that controls how big each update step is during training.
- Minimum
- A minimum is a low point of the loss function where the model has less error than nearby parameter values.
Common Mistakes to Avoid
- Thinking gradient descent always finds the absolute best answer. This is wrong because complex loss landscapes can have local minima, flat regions, or other challenges.
- Using a very large learning rate to make training faster. This is wrong because the updates may overshoot the minimum and make the loss bounce around or increase.
- Using a very small learning rate without checking progress. This is wrong because the model may improve so slowly that training takes too long.
- Moving in the same direction as the gradient. This is wrong because the gradient points uphill, while gradient descent must move in the opposite direction to reduce loss.
Practice Questions
- 1 A model has one parameter w = 6, learning rate α = 0.2, and gradient dL/dw = 4. Use w_new = w_old - α dL/dw to find the new value of w.
- 2 A model's loss values after five training steps are 20, 14, 9, 7, and 6. What is the total decrease in loss from the first step to the fifth step?
- 3 A student increases the learning rate and notices that the loss starts jumping up and down instead of steadily decreasing. Explain what is likely happening and how the student could fix it.