Neural network activation functions decide how each neuron transforms its weighted input into an output. They make deep networks nonlinear, which allows models to learn complex patterns instead of only linear relationships. This cheat sheet helps students compare common activations, remember their formulas, and understand when each one is useful.
It is especially helpful for studying feedforward networks, convolutional networks, transformers, and classification models.
The most important ideas are output range, derivative behavior, saturation, and computational cost. Sigmoid and tanh are smooth but can suffer from vanishing gradients when inputs are very large or very negative. ReLU and its variants are simple and efficient, while GELU and Swish are smooth activations often used in modern deep learning.
Softmax is used at the output layer for multiclass classification because it converts logits into probabilities that sum to 1.
Key Facts
- Sigmoid is defined as sigmoid(x) = 1 / (1 + exp(-x)) and outputs values between 0 and 1.
- The derivative of sigmoid is sigmoid'(x) = sigmoid(x)(1 - sigmoid(x)), which becomes small when sigmoid(x) is near 0 or 1.
- Tanh is defined as tanh(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x)) and outputs values between -1 and 1.
- ReLU is defined as ReLU(x) = max(0, x), which is fast and helps reduce vanishing gradients for positive inputs.
- Leaky ReLU is defined as LeakyReLU(x) = x if x > 0, and alpha x if x <= 0, where alpha is usually a small value such as 0.01.
- GELU is often approximated by GELU(x) = 0.5x(1 + tanh(sqrt(2/pi)(x + 0.044715x^3))) and is common in transformer models.
- Softmax is defined as softmax(z_i) = exp(z_i) / sum_j exp(z_j), converting a vector of logits into class probabilities.
- For numerical stability, softmax should be computed as exp(z_i - max(z)) / sum_j exp(z_j - max(z)).
Vocabulary
- Activation function
- A function applied to a neuron's pre-activation value to produce the neuron's output.
- Logit
- A raw, unnormalized model output value before applying sigmoid or softmax.
- Saturation
- A condition where an activation function's output changes very little even when its input changes.
- Vanishing gradient
- A training problem where gradients become extremely small, making earlier layers learn very slowly.
- Nonlinearity
- A property that allows a neural network to model relationships that cannot be represented by a single linear function.
- Dying ReLU
- A problem where a ReLU neuron outputs 0 for all inputs and receives no gradient to recover.
Common Mistakes to Avoid
- Using sigmoid in many hidden layers, because its gradients can become very small when activations saturate near 0 or 1.
- Applying softmax before cross-entropy loss when the library expects raw logits, because many implementations combine softmax and cross-entropy internally for numerical stability.
- Forgetting that ReLU has zero gradient for x < 0, because neurons can stop updating if their inputs remain negative.
- Treating activation choice as independent of the task, because output activations should match the target type, such as sigmoid for binary probabilities and softmax for multiclass probabilities.
- Computing softmax directly with large logits, because exp(z) can overflow and should be stabilized by subtracting max(z).
Practice Questions
- 1 Compute sigmoid(0), sigmoid(2), and sigmoid(-2) to three decimal places.
- 2 For LeakyReLU with alpha = 0.01, compute the outputs for x = -5, x = 0, and x = 3.
- 3 Given logits z = [1, 2, 4], compute the softmax probabilities to three decimal places using the stable softmax method.
- 4 Explain why ReLU often trains faster than sigmoid in hidden layers, and describe one situation where ReLU can fail.
Understanding Neural Network Activation Functions Reference
During training, an activation function affects more than the value passed forward. Its slope affects the error signal passed backward through the network. Backpropagation uses the chain rule, so gradients from later layers are multiplied by activation derivatives from earlier layers.
Many small derivatives multiplied together can make early layers learn extremely slowly. This is why saturation is a serious practical issue, not just a graph shape to memorize. Sigmoid and tanh have their largest slopes near the middle of their ranges.
Inputs pushed far from that middle produce weak updates. Tanh is often easier to optimize than sigmoid in hidden layers because its outputs are centered around zero. This can make weight updates less biased in one direction.
ReLU avoids saturation on its positive side, but it has a different failure mode. A neuron with a negative input produces zero output and receives no gradient through the usual ReLU rule. If this happens for nearly every training example, that neuron can become permanently inactive.
This is called a dead ReLU. Large negative biases, an overly high learning rate, or poor initialization can increase the risk. Leaky ReLU keeps a small negative slope, so inactive-looking neurons still receive a small learning signal.
GELU behaves more gradually. Instead of applying a hard cutoff, it smoothly reduces small or negative values while retaining useful information. This smooth behavior is one reason it appears often in transformer architectures.
The final layer needs an activation that matches the meaning of the target labels. For a binary decision, such as whether an email is spam, one sigmoid output can represent the model's confidence for one class. For several independent labels, such as tags on an image, separate sigmoid outputs allow more than one tag to be present.
Softmax is different because its outputs compete with one another. Raising one class score lowers the relative probability of the others. It fits tasks with one correct class, such as recognizing one digit from zero through nine.
The values sent into softmax are called logits. They are raw scores, not probabilities. In code, subtracting the largest logit before exponentiation keeps the calculation safe from overflow while leaving the final probabilities unchanged.
Activation selection is usually guided by evidence from training, not by a single universal rule. ReLU or a related variant is a common starting point for ordinary hidden layers. GELU is often chosen when reproducing transformer designs.
Sigmoid and tanh remain useful when their bounded ranges match the problem, such as gates in some recurrent networks. Students should pay close attention to the derivative at zero, behavior for large positive and negative inputs, output centering, and whether an activation can produce exact zeros.
When debugging a model, inspect activation values and gradient sizes layer by layer. A network can have a correct loss function yet still train poorly because activations are saturated, neurons are dead, or values grow too large.