Gradient descent is an optimization method used to train many machine learning models by reducing a loss function step by step. This cheat sheet helps students connect the geometry of a loss surface with the algebra of parameter updates. It is useful for understanding linear regression, logistic regression, neural networks, and many other models.
The goal is to make the update rule, learning rate choices, and convergence behavior easy to reference while solving problems or reading code.
The core idea is to move parameters in the direction opposite the gradient because the gradient points toward steepest increase. The basic update is theta_new = theta_old - alpha * gradient, where alpha is the learning rate. Batch, stochastic, and mini-batch gradient descent differ in how much data is used to estimate each gradient.
Good optimization depends on scaling features, choosing a stable learning rate, and checking whether the loss actually decreases over time.
Key Facts
- Gradient descent updates parameters using theta_new = theta_old - alpha * grad J(theta), where J(theta) is the loss function.
- The gradient grad J(theta) points in the direction of steepest increase, so subtracting it moves toward lower loss.
- For one parameter, the update rule is theta_new = theta_old - alpha * dJ/dtheta.
- For mean squared error in linear regression, J(theta) = (1/m) * sum((y_hat_i - y_i)^2) or sometimes J(theta) = (1/(2m)) * sum((y_hat_i - y_i)^2).
- A learning rate that is too large can make the loss oscillate or diverge, while a learning rate that is too small can make training very slow.
- Batch gradient descent uses all m training examples per update, stochastic gradient descent uses 1 example per update, and mini-batch gradient descent uses a small subset per update.
- Convergence is often checked by monitoring whether J(theta) decreases and whether the change in loss or parameters becomes very small.
- Feature scaling helps gradient descent because parameters then have more balanced gradients, reducing zigzag paths across narrow loss contours.
Vocabulary
- Loss function
- A function J(theta) that measures how wrong a model's predictions are for a given set of parameters.
- Gradient
- A vector of partial derivatives that shows how the loss changes as each parameter changes.
- Learning rate
- The step size alpha that controls how far parameters move during each gradient descent update.
- Parameter
- A model value such as a weight or bias that is adjusted during training to reduce loss.
- Convergence
- The condition where training has mostly stabilized because the loss or parameter updates change very little.
- Mini-batch
- A small subset of training examples used to compute an approximate gradient for one update.
Common Mistakes to Avoid
- Using the wrong sign in the update is wrong because theta_new = theta_old + alpha * grad J(theta) moves uphill for minimization instead of downhill.
- Choosing a learning rate without checking the loss is risky because a large alpha can cause divergence and a tiny alpha can hide progress.
- Comparing gradient descent runs on unscaled features can be misleading because large feature ranges can dominate gradients and slow convergence.
- Assuming every loss surface has one minimum is wrong because many models, especially neural networks, can have nonconvex surfaces with local minima and saddle points.
- Stopping only after a fixed number of iterations can be poor practice because the loss may still be decreasing quickly or may have stopped improving much earlier.
Practice Questions
- 1 For J(theta) with dJ/dtheta = 8 at theta = 3 and alpha = 0.1, compute theta_new using theta_new = theta - alpha * dJ/dtheta.
- 2 A model has theta = [2, -1], gradient = [4, -6], and alpha = 0.05. Compute the updated parameter vector.
- 3 A training run has losses 12.0, 7.5, 5.1, 5.0, 5.0 over five checks. What does this suggest about convergence, and what might you inspect next?
- 4 Explain why mini-batch gradient descent is often preferred over full batch gradient descent for very large datasets.
Understanding Gradient Descent Reference
A gradient is more than a list of slopes. Each entry tells how sensitive the loss is to one parameter while the other parameters are held fixed. If a weight has a large positive gradient, increasing that weight would raise the loss quickly.
If its gradient is near zero, a small change to that weight has little immediate effect. In a model with thousands or millions of weights, the gradient gives one direction through a very high-dimensional space.
Calculus makes this possible because derivatives provide a local approximation. The approximation is trustworthy only for a sufficiently small step, which is one reason step size matters.
For linear regression, each prediction error contributes to the gradient. A data point with a prediction far from its target usually creates a stronger push than a point predicted accurately. The sign of the error determines which way a weight should move.
Input values matter too. A large input can produce a large contribution to a weight gradient. This explains why unscaled features can cause trouble.
If one feature is measured in tiny fractions and another in millions, their updates can have very different sizes. Standardising inputs so that they have similar ranges often makes training easier to control.
Neural networks use the chain rule to calculate gradients efficiently. The model first performs a forward pass, producing predictions and a loss. It then works backward from the loss through each layer.
This process is called backpropagation. Each connection receives a signal showing how changing it would affect the final loss. The calculation does not mean every parameter is equally useful or equally certain.
Some gradients can become extremely small in deep networks, making early layers learn slowly. Others can become extremely large, causing unstable updates. Techniques such as careful weight initialisation, suitable activation functions, gradient clipping, and normalisation are used to reduce these problems.
A falling training loss is useful evidence, but it is not the whole story. Students should compare training performance with validation performance on examples not used for updates. If training loss keeps falling while validation loss rises, the model is memorising details instead of learning patterns that generalise.
Regularisation can discourage overly complex parameter values. Mini-batches add some randomness because each update sees only part of the data. That noise can make the loss graph look uneven, even when learning is progressing.
It can sometimes help the model move away from flat regions or saddle points, where the gradient is small but the parameters are not at the best solution. When reading training logs, look for overall trends across many updates, not just one unusually good or bad batch.