The encoder-decoder architecture is a neural network pattern for handling tasks where input and output can have different lengths. This setup uses two separate neural networks.
The encoder reads the input, such as a full sentence. Its job is to absorb and compress all the information from the input into a single, fixed-size vector. This vector is called the context vector or thought vector. Think of it like reading a paragraph and distilling its whole meaning into a dense summary.
The decoder uses this context vector to generate the output. The output could be a translation, a summary, a reply, or any sequence that may be a different length or in a different format from the input.
This design allows neural networks to handle tasks where input and output aren’t the same length or even the same “type.”
Most classic neural networks handle only fixed-size inputs and outputs. But many real tasks, especially in natural language processing (NLP), need more flexibility.
Machine translation is a good example. Translating “How are you?” (3 words) into French gives “Comment ça va ?” (4 words). The output doesn’t always match the input in length or word order.
Trying to map input to output one word at a time misses context and grammar. Some translations need the full sense of the sentence before choosing the right words. Encoder-decoder models solve this by letting the network process the whole input before producing any output.
The encoder-decoder architecture is made for:
The encoder starts with a sequence of input tokens, such as words in a sentence. It processes these tokens one by one, updating an internal state as it goes. This state is usually managed by a recurrent neural network (RNN), which “remembers” what it has seen so far.
When the encoder finishes reading the entire input sequence, the final value of its internal state becomes the context vector. This vector is a compressed representation of the whole input.
For example, encoding “hello world”:
The encoder does not know yet what the output will be. Its job is only to produce the best possible summary of the input.
The decoder is another neural network, usually another RNN. It starts with the context vector from the encoder. At each step, it generates one output token.
The decoder uses its own internal state, plus the context from the encoder, and often the previous output it produced. With this information, it chooses the next output token.
For English-to-French translation, for example:
This process allows the decoder to produce outputs of any length, only stopping when the task is finished.
Let’s see how this works when translating “cat sleeps” into French: “le chat dort.”
Encoding:
Decoding:
At each step, the decoder’s output draws from both its own state and the original context. This helps ensure the translation stays grammatical and meaningful.
Let's use a simple problem: reversing a sequence of numbers. For input [1, 2, 3], the output should be [3, 2, 1]. This demonstrates every part of the encoder-decoder setup.
We’ll use PyTorch for this example. The code is minimal and focused on the idea.
import torch
import torch.nn as nn
# Parameters
INPUT_SIZE = 10 # Numbers 0–9
HIDDEN_SIZE = 16
OUTPUT_SIZE = 10 # Same as input
SEQ_LEN = 3
# Encoder: RNN that reads the input
class Encoder(nn.Module):
def __init__(self):
super().__init__()
self.embedding = nn.Embedding(INPUT_SIZE, HIDDEN_SIZE)
self.rnn = nn.GRU(HIDDEN_SIZE, HIDDEN_SIZE)
def forward(self, input_seq):
embedded = self.embedding(input_seq).unsqueeze(1)
outputs, hidden = self.rnn(embedded)
return hidden
# Decoder: RNN that writes the output, one token at a time
class Decoder(nn.Module):
def __init__(self):
super().__init__()
self.embedding = nn.Embedding(OUTPUT_SIZE, HIDDEN_SIZE)
self.rnn = nn.GRU(HIDDEN_SIZE, HIDDEN_SIZE)
self.fc = nn.Linear(HIDDEN_SIZE, OUTPUT_SIZE)
def forward(self, input_token, hidden):
embedded = self.embedding(input_token).unsqueeze(0)
output, hidden = self.rnn(embedded, hidden)
logits = self.fc(output.squeeze(0))
return logits, hidden
# Example: run once (no training)
encoder = Encoder()
decoder = Decoder()
input_seq = torch.tensor([1, 2, 3], dtype=torch.long)
target_seq = torch.tensor([3, 2, 1], dtype=torch.long)
# Encode
context = encoder(input_seq)
# Decode step by step
decoded_tokens = []
decoder_input = torch.tensor([0]) # Start token (use 0 here)
hidden = context
for i in range(SEQ_LEN):
logits, hidden = decoder(decoder_input, hidden)
prediction = logits.argmax(1)
decoded_tokens.append(prediction.item())
decoder_input = prediction
print("Predicted output:", decoded_tokens)
This code uses two small RNNs: one encodes the input sequence into a context vector, and the other decodes that context to produce the output sequence. There’s no training here, so the outputs will be random, but it shows how information flows from encoder to decoder.
The classic encoder-decoder model squeezes all the input into a single fixed-length context vector. For short or simple sentences, this works. But as the input grows longer or more complex, it’s easy for important details to vanish. This “bottleneck” limits the model’s performance.
Modern architectures address this by letting the decoder look back at the whole input using mechanisms like "attention." The encoder-decoder idea is the foundation. From here, we'll see how to overcome its biggest limits and handle even more challenging tasks.
Write a function that takes a list of numbers (such as [3, 1, 4]) and uses a pre-defined encoder-decoder model to reverse the sequence. Run your function with several different input lists to observe how the encoder compresses the input and the decoder reconstructs the output. Reflect on how the architecture changes with different sequence lengths.
Coming up on Day 24: Attention Mechanism: The Core Intuition