Gradient descent is a method for finding the input values that make a function as small as possible. It is used in applied math, data science, physics, engineering, and machine learning. This cheat sheet helps students connect slopes, derivatives, and vectors to a practical optimization algorithm.
It is useful when a formula is too complex to minimize by simple algebra.
Key Facts
- To minimize a one-variable function f(x), gradient descent uses the update x_new = x_old - alpha f'(x_old).
- For a multivariable function f(x, y), the gradient is grad f = <partial f/partial x, partial f/partial y>.
- Gradient descent moves in the negative gradient direction because the gradient points in the direction of steepest increase.
- The learning rate alpha controls step size, and a common update rule is theta_new = theta_old - alpha grad J(theta_old).
- A critical point occurs when grad f = 0, but it may be a minimum, maximum, or saddle point.
- Convergence means the updates become very small, often checked by |theta_new - theta_old| < tolerance or |grad J| < tolerance.
- A convex function has one global minimum, so gradient descent is less likely to get stuck at a non-best local minimum.
- If alpha is too large, the algorithm can overshoot or diverge, and if alpha is too small, the algorithm can be very slow.
Vocabulary
- Objective function
- The function being minimized or maximized in an optimization problem.
- Gradient
- A vector of partial derivatives that points in the direction where a multivariable function increases fastest.
- Learning rate
- The positive number alpha that controls how large each gradient descent step is.
- Convergence
- The process of an iterative method getting closer to a stable answer or minimum.
- Local minimum
- A point where the function value is lower than nearby points but not necessarily the lowest value overall.
- Convex function
- A function shaped so that any local minimum is also the global minimum.
Common Mistakes to Avoid
- Using x_new = x_old + alpha f'(x_old) when minimizing is wrong because it moves in the direction of increase, not decrease.
- Choosing a learning rate that is too large is wrong because the steps can jump over the minimum and make the values grow instead of shrink.
- Assuming grad f = 0 always means a minimum is wrong because the point could be a maximum or a saddle point.
- Stopping after one or two iterations is wrong because gradient descent usually needs repeated updates until the change or gradient is small.
- Forgetting that the gradient is a vector in multivariable problems is wrong because each variable needs its own update using its partial derivative.
Practice Questions
- 1 For f(x) = x^2 - 6x + 10, start at x = 0 with alpha = 0.1. Compute one gradient descent update.
- 2 For f(x) = (x - 4)^2, start at x = 10 with alpha = 0.25. Compute two gradient descent updates.
- 3 For J(a, b) = a^2 + 3b^2, find grad J at (2, -1) and write the update formula using learning rate alpha.
- 4 Explain why moving in the negative gradient direction helps minimize a function, and describe what can go wrong if the learning rate is too large.
Understanding Gradient Descent & Optimization
An objective function is a score for a proposed solution. Lower scores represent solutions that better meet the goal. In a line fitting task, the parameters may be the slope and intercept of a line.
The objective can measure the total squared difference between predicted values and measured values. In physics, parameters might represent a spring constant, a drag coefficient, or an initial velocity. The algorithm changes these adjustable parameters repeatedly.
Each change is judged by whether it lowers the score. This turns a vague task, such as matching a model to data, into a sequence of numerical decisions.
The gradient is based on local slope information. In one dimension, a derivative tells how the function changes near the current input. A positive derivative means that moving right raises the function nearby, so a move left is useful for minimization.
With several parameters, each partial derivative describes sensitivity to one parameter while the others are held fixed. Large gradient components show parameters that currently have a strong effect on the objective. Small components show directions with less immediate effect.
On a contour map, the gradient points across contour lines toward faster increase. Moving opposite to it crosses toward lower contours. This picture helps students connect vector direction to the behavior of a surface.
Real optimization is often harder than the smooth bowl shown in basic examples. Parameters can have very different scales. One parameter might be measured in thousands while another is less than one.
This can create a long, narrow valley. Ordinary steps may bounce from side to side instead of heading efficiently downhill. Rescaling input data or standardizing features often improves this problem.
Many machine learning systems calculate gradients from a small random group of training examples rather than the whole data set each time. These estimates contain noise, so the objective may rise on some individual steps even when the overall trend is downward. Methods such as momentum use part of the previous movement to reduce zigzagging.
A small gradient does not automatically mean that a useful answer has been found. A flat region can slow progress, and a saddle point can have uphill directions mixed with downhill directions. Checking the objective value over many iterations gives more information than checking only one update.
Students should watch for values that decrease steadily, values that oscillate, or values that grow without bound. They should separate training performance from performance on new data. A model can obtain a very low training objective by fitting random noise, then make poor predictions elsewhere.
When working problems by hand, state the current point, calculate the derivative or gradient there, choose a step size, update every parameter, and repeat carefully. Keeping these stages separate prevents most sign and arithmetic errors.