Backpropagation is the main method many neural networks use to learn from mistakes. A neural network makes a prediction, compares it with the correct answer, then adjusts its internal numbers to do better next time. This matters because it helps power tools such as image recognition, speech assistants, recommendation systems, and game-playing AI.
The basic idea is simple: send information forward, measure the error, then send correction signals backward.
Key Facts
- Forward pass: inputs move through the network to produce a prediction.
- Loss function: loss = measure of how wrong the prediction is.
- Error signal: error = predicted value - true value for a simple output.
- Gradient: a gradient tells how much a small change in a weight changes the loss.
- Weight update: new weight = old weight - learning rate × gradient.
- Learning rate controls step size, so too large can overshoot and too small can learn slowly.
Vocabulary
- Neural network
- A computing system made of connected layers that transform input data into predictions.
- Weight
- A weight is a number on a connection that controls how strongly one neuron affects another.
- Loss function
- A loss function is a formula that measures how far a model prediction is from the correct answer.
- Gradient
- A gradient is the direction and size of change that shows how to adjust a weight to reduce loss.
- Learning rate
- The learning rate is a chosen number that controls how big each weight update step will be.
Common Mistakes to Avoid
- Thinking backpropagation is the same as artificial intelligence, but it is only one learning algorithm used to train many neural networks.
- Updating weights in the direction of the gradient, but gradient descent subtracts the gradient because the goal is to reduce the loss.
- Using a learning rate without thinking about scale, because a rate that is too large can make training unstable and a rate that is too small can make training very slow.
- Ignoring the forward pass, but backpropagation needs the forward pass values to compute errors and gradients correctly.
Practice Questions
- 1 A model predicts 0.80 for a correct target of 1.00. Using error = predicted value - true value, what is the error?
- 2 A weight is 0.50, the gradient is 0.20, and the learning rate is 0.10. Using new weight = old weight - learning rate × gradient, what is the new weight?
- 3 A neural network keeps making very large jumps in loss, sometimes getting better and sometimes much worse. Explain why lowering the learning rate might help.