A recurrent neural network, or RNN, is a type of artificial neural network designed to work with data that comes in a sequence. This matters because many real-world problems depend on order, such as words in a sentence, notes in music, or temperature readings over time. Unlike a basic neural network that treats each input separately, an RNN keeps a memory of what came before.
That memory helps it make better predictions about what comes next.
Key Facts
- An RNN processes a sequence one step at a time, such as x1, x2, x3, ...
- The hidden state stores information from earlier steps in the sequence.
- A simple RNN update can be written as h_t = tanh(W_x x_t + W_h h_(t-1) + b).
- The output at a time step can be written as y_t = W_y h_t + c.
- RNNs are useful for text prediction, speech recognition, translation, music generation, and time-series forecasting.
- Long short-term memory networks and gated recurrent units improve RNNs by helping them remember important information for longer.
Vocabulary
- Recurrent Neural Network
- A neural network that processes ordered data by passing information from one step of the sequence to the next.
- Sequence
- A set of data points where the order matters, such as words in a sentence or values measured over time.
- Hidden State
- The internal memory of an RNN that carries information from previous inputs to help process the current input.
- Time Step
- One position in a sequence where the RNN reads an input and updates its hidden state.
- Training
- The process of adjusting a model's weights so its predictions become closer to the correct answers.
Common Mistakes to Avoid
- Treating an RNN like a regular feedforward network is wrong because an RNN reuses information from earlier time steps instead of processing each input independently.
- Ignoring the order of inputs is wrong because changing the order of words, notes, or measurements can change the meaning of the sequence.
- Assuming the hidden state remembers everything perfectly is wrong because basic RNNs can forget older information, especially in long sequences.
- Confusing training with prediction is wrong because training updates the model's weights, while prediction uses learned weights to produce an output.
Practice Questions
- 1 A sentence has 8 words, and an RNN processes one word per time step. How many time steps are needed to process the full sentence?
- 2 A simple RNN has an input vector with 4 numbers and a hidden state with 6 numbers. The matrix W_x connects the input to the hidden state. How many weights are in W_x?
- 3 A model must predict the next word in a sentence. Explain why an RNN can use earlier words in the sentence to make a better prediction than a model that only sees the current word.