Previously, on Day 17: Explained the structure and function of neural networks, including neurons, layers, weights, biases, and activation functions, and how neural networks are trained to solve natural language processing tasks.
Human language is structured as a sequence. The meaning of a sentence depends on the order of its words:
Feedforward neural networks, which we've seen so far, treat each input as independent from the others. These models can classify an image or predict a single label, but they have no way to remember earlier inputs. If you process a sentence word by word, a feedforward network processes each word as if it's unrelated to what came before.
This becomes a serious limitation for tasks like machine translation, speech recognition, or conversational context. In language, the order of words and the context from earlier words are essential. We need models that can remember what has already happened and use that memory to interpret each new word.
A Recurrent Neural Network (RNN) is designed for sequential data. RNNs process data one element at a time—think one word at a time—while carrying a short-term memory, called the hidden state, from one step to the next.
Unlike feedforward networks, each step in an RNN passes information forward through this hidden state. The network can "remember" what it has seen and use that memory when processing new inputs.
This memory makes RNNs a much better fit for sequences like sentences, where the meaning of a word depends on what came before.
Imagine reading a sentence one word at a time from left to right. You don't reset your memory at each word; you remember what you’ve read so far. RNNs are built with this same idea.
To make training possible, we unroll the RNN in time:
At each step, the RNN cell:
This chained process lets information—the hidden state—move forward through the sequence and capture context up to the current position.
Inside an RNN cell, the computation is similar to a layer in a regular neural network, but with one key difference: it takes two inputs at each timestep—
The core update at each time step ( t ) is:
[
h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b_h)
]
Where:
The new hidden state ( h_t ) carries information from both the new input and everything summarized up to the previous step.
Here's a minimal PyTorch example showing how an RNN processes a sequence. This code:
import torch
import torch.nn as nn
# Example sequence: batch size 1, sequence length 5, input size 1
sequence = torch.tensor([[[1.0]], [[2.0]], [[3.0]], [[4.0]], [[5.0]]])
input_size = 1 # Input dimension at each step
hidden_size = 2 # Hidden state size
rnn = nn.RNN(input_size, hidden_size, batch_first=False)
# Initialize hidden state: (num_layers, batch_size, hidden_size)
h_t = torch.zeros(1, 1, hidden_size)
print("Step | Input | Hidden State")
print("-------------------------")
for i, x_t in enumerate(sequence):
out, h_t = rnn(x_t.unsqueeze(0), h_t) # Add batch/time axes
print(f" {i+1} | {x_t.item()} | {h_t.squeeze(0).detach().numpy()}")
At each step, the RNN's hidden state reflects not just the current number, but also what it's seen so far. This is what gives the RNN its memory of the sequence.
Basic, or "vanilla," RNNs can remember context for short sequences, but they struggle with longer ones.
The challenge comes from training. As the RNN updates its weights, important signals from early in the sequence can quickly shrink to near zero (vanish) or grow out of control (explode). This is called the vanishing or exploding gradient problem. In practice, vanilla RNNs often forget information from more than a few steps back.
To fix this, researchers developed enhanced RNNs, such as LSTMs (Long Short-Term Memory networks) and GRUs (Gated Recurrent Units). These are specially designed to remember important information for much longer, allowing models to capture context even when it appears many words back in a sentence or conversation.
Understanding basic RNNs gives you the foundation to appreciate why these more advanced architectures are needed—and how they power real-world NLP systems, where meaning can depend on something far earlier in the sequence.
Choose a short word (such as 'cat') and represent each character as a simple number (e.g., c=1, a=2, t=3). Using the update rule h_t = tanh(0.5 * x_t + 0.8 * h_{t-1}) with h_0 = 0, manually compute and record the hidden state for each character in order. Write down the value of h after processing each character to observe how the hidden state accumulates information.
Coming up on Day 19: The Vanishing Gradient Problem