A recurrent neural network, or RNN, is a type of artificial neural network designed to work with data that comes in a sequence. This matters because many real-world problems depend on order, such as words in a sentence, notes in music, or temperature readings over time. Unlike a basic neural network that treats each input separately, an RNN keeps a memory of what came before.
That memory helps it make better predictions about what comes next.
Understanding AI & Machine Learning: Recurrent Neural Networks Explained
An RNN uses the same set of learned weights at every position in a sequence. This is important because a sentence may have five words or fifty, yet the network can apply one repeated rule to each word. At each step, it combines the new input with its current hidden state.
The hidden state is a list of numbers, not a stored copy of every earlier item. It is a compressed summary.
During training, the network learns which details should affect that summary. For language, it may keep clues about grammar, topic, or a word that changes the meaning of a later word.
Training an RNN means showing it many examples and measuring how far its predictions are from the correct answers. The error is sent backward through every step of the sequence so the weights can be adjusted. This method is called backpropagation through time.
It can be difficult when sequences are long. Small changes in the error signal may shrink until they are nearly zero. This is called the vanishing gradient problem.
Errors can grow too large as well, causing unstable learning. These problems explain why a basic RNN often struggles to connect information that is far apart, such as a subject near the start of a long sentence with a verb near the end.
LSTM networks and GRU networks were designed to handle this weakness. They use gates, which are small learned controls that decide what information to keep, update, or forget. An LSTM has a separate cell state that can carry selected information across many steps.
A GRU uses a simpler gate design with fewer parts. Neither model remembers everything perfectly. They learn to protect information that helps reduce training error.
In a translation task, this may include the meaning of an early word until the model reaches the matching word in another language. In sensor data, it may include a gradual trend rather than a single noisy reading.
Students meet sequence models in keyboard word suggestions, speech to text, captions, forecasts of demand, and systems that flag unusual patterns in machine data. The output depends heavily on the training examples. A model trained on limited speech accents may work less well for other speakers.
A forecast can fail when conditions change sharply, such as during an unusual storm or a sudden supply shortage. When learning RNNs, pay attention to the difference between model memory and human memory. The hidden state is a changing numerical signal.
Focus on the flow of information across time, the shared weights, and the reason gates help. Those ideas make later topics such as attention and transformers easier to understand.
Key Facts
- An RNN processes a sequence one step at a time, such as x1, x2, x3, ...
- The hidden state stores information from earlier steps in the sequence.
- A simple RNN update can be written as h_t = tanh(W_x x_t + W_h h_(t-1) + b).
- The output at a time step can be written as y_t = W_y h_t + c.
- RNNs are useful for text prediction, speech recognition, translation, music generation, and time-series forecasting.
- Long short-term memory networks and gated recurrent units improve RNNs by helping them remember important information for longer.
Vocabulary
- Recurrent Neural Network
- A neural network that processes ordered data by passing information from one step of the sequence to the next.
- Sequence
- A set of data points where the order matters, such as words in a sentence or values measured over time.
- Hidden State
- The internal memory of an RNN that carries information from previous inputs to help process the current input.
- Time Step
- One position in a sequence where the RNN reads an input and updates its hidden state.
- Training
- The process of adjusting a model's weights so its predictions become closer to the correct answers.
Common Mistakes to Avoid
- Treating an RNN like a regular feedforward network is wrong because an RNN reuses information from earlier time steps instead of processing each input independently.
- Ignoring the order of inputs is wrong because changing the order of words, notes, or measurements can change the meaning of the sequence.
- Assuming the hidden state remembers everything perfectly is wrong because basic RNNs can forget older information, especially in long sequences.
- Confusing training with prediction is wrong because training updates the model's weights, while prediction uses learned weights to produce an output.
Practice Questions
- 1 A sentence has 8 words, and an RNN processes one word per time step. How many time steps are needed to process the full sentence?
- 2 A simple RNN has an input vector with 4 numbers and a hidden state with 6 numbers. The matrix W_x connects the input to the hidden state. How many weights are in W_x?
- 3 A model must predict the next word in a sentence. Explain why an RNN can use earlier words in the sentence to make a better prediction than a model that only sees the current word.