Activation functions are small mathematical rules that help artificial neural networks make decisions. A neuron first combines inputs, weights, and a bias into one number, then the activation function transforms that number into an output. This matters because without activation functions, a neural network would behave like one big linear equation.
With them, AI systems can learn curved patterns, categories, images, speech, and many other complex relationships.
The basic flow is input values go in, weights adjust their importance, a bias shifts the result, and an activation function decides what signal to pass forward. Common activation functions include sigmoid, tanh, ReLU, and softmax, each useful in different parts of a model. During training, the network uses errors and gradients to adjust weights so the outputs improve over time.
Choosing the right activation function can make learning faster, more stable, and more accurate.
Key Facts
- Neuron input sum: z = w1x1 + w2x2 + ... + b
- Activation output: a = f(z)
- Sigmoid function: f(z) = 1 / (1 + e^(-z)), and its output is between 0 and 1
- ReLU function: f(z) = max(0, z), so negative inputs become 0
- Tanh function: f(z) = (e^z - e^(-z)) / (e^z + e^(-z)), and its output is between -1 and 1
- Softmax converts scores into class probabilities: p_i = e^(z_i) / sum(e^(z_j))
Vocabulary
- Activation function
- A mathematical rule that transforms a neuron's input sum into an output signal.
- Artificial neuron
- A simple computing unit in a neural network that combines inputs, weights, a bias, and an activation function.
- Weight
- A number that controls how strongly an input affects a neuron's final result.
- Bias
- A number added to the weighted input sum to shift when a neuron becomes active.
- Gradient
- A value that tells the learning algorithm how a small change in a parameter affects the model's error.
Common Mistakes to Avoid
- Forgetting the bias term is wrong because the neuron may not be able to shift its activation boundary to fit the data well.
- Using only linear activation functions is wrong because stacking linear layers still produces a linear model, which cannot learn many complex patterns.
- Thinking ReLU gives negative outputs is wrong because ReLU changes every negative input to 0 and leaves positive inputs unchanged.
- Using softmax for a single yes-or-no output is usually wrong because sigmoid is better suited for one binary probability, while softmax is designed to compare multiple classes.
Practice Questions
- 1 A neuron has inputs x1 = 2 and x2 = -1, weights w1 = 0.5 and w2 = 3, and bias b = 1. Find z = w1x1 + w2x2 + b, then find the ReLU output.
- 2 Use the sigmoid approximation f(z) = 1 / (1 + e^(-z)). If z = 0, what is the sigmoid output? If z = 2 and e^(-2) is about 0.135, what is the sigmoid output to three decimal places?
- 3 A hidden layer in a neural network uses no activation function, only weighted sums. Explain why this limits the kinds of patterns the network can learn, even if it has many layers.