Understanding sequences, like sentences, depends on how words relate to each other. In language, a word's meaning can depend on words that come much earlier or later in the sentence.
Traditional sequence models, such as RNNs (Recurrent Neural Networks), process sequences one word at a time. Each step updates a memory that carries information forward. Passing information from the start of a sentence to the end requires many steps, which makes it easy to lose details—especially in long sentences.
CNNs (Convolutional Neural Networks) look at groups of nearby words to spot patterns. But connecting information between distant words, like the beginning and end of a sentence, takes stacking many CNN layers. This quickly becomes inefficient.
Both of these approaches struggle when long-range relationships matter. For instance, in the sentence “The animal, which the farmer chased, ran away,” the subject “animal” pairs with the verb “ran away,” but they're separated by several words. We need a model that gives every word direct access to all others, so it can use whatever context matters, even across long distances.
Self-attention solves this problem. It lets every word see and weigh all the other words in the sequence simultaneously.
Self-attention is a process where each word looks at all other words in the sequence to decide which are important for its own representation. Instead of marching step by step, every word gets information from everywhere else, all at once.
Think of reading a sentence. For each word, you glance at the rest of the sentence to check what changes its meaning. Some words might barely matter, others might be essential.
Analogy: Imagine a group meeting. Each person (word) listens to everyone else before forming their opinion. You might care a lot about what one person says, and barely note another. The weight you give to each speaker shapes your final thought.
Self-attention uses three parts for every input word: a query, a key, and a value. These are just different views of the same word, computed as simple linear projections.
Let’s use an example:
"The hungry cat chased the mouse."
When focusing on “cat”:
The query, key, and value for each word are just different mathematical transformations (matrix multiplications) of its word embedding—the basic numeric representation of a word.
Let's walk through a concrete example using tiny vectors, so you can see every math step.
Suppose our vocabulary consists of just three words: A, B, and C. Each word is represented as a 2-component vector.
Let’s define:
Our sequence is [A, B, C].
To keep things simple, we’ll set the query, key, and value for each word equal to the input vector. (Normally, you’d multiply by separate matrices. Here, we use the identity matrix, meaning nothing changes.)
Q = [[1, 0], # A’s query
[0, 1], # B’s query
[1, 1]] # C’s query
K = [[1, 0], # A’s key
[0, 1], # B’s key
[1, 1]] # C’s key
V = [[1, 0], # A’s value
[0, 1], # B’s value
[1, 1]] # C’s value
For each word, calculate the dot product between its query and the keys of every word. This gives a score for how much each word might matter.
For A:
A’s attention scores: [1, 0, 1]
Repeat for B and C:
Softmax is a math function that converts scores to positive numbers that sum to 1. This turns the raw scores into attention weights.
For A: softmax([1, 0, 1])
First, exponentiate: exp(1)=2.718, exp(0)=1
Sum = 2.718 + 1 + 2.718 = 6.436
Weights:
So A pays 42% attention to itself, 16% to B, 42% to C.
Each word calculates its new representation as a weighted sum of all value vectors (one for each word), using its attention weights.
For A:
Repeat this for B and C to get their new context-aware representations.
Now, every output vector contains a blend of information from the whole sequence, mixed according to what each word considers important.
import numpy as np
# Step 1: inputs (3 words, each as 2D vector)
X = np.array([
[1., 0.], # Word A
[0., 1.], # Word B
[1., 1.] # Word C
])
# Step 2: queries, keys, values (use identity - no change)
Q = X
K = X
V = X
# Step 3: attention scores (dot product Q and K.T)
scores = Q @ K.T
# Step 4: softmax (row-wise)
def softmax(x):
e_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
return e_x / e_x.sum(axis=-1, keepdims=True)
weights = softmax(scores)
# Step 5: attention output (weights @ V)
output = weights @ V
print("Attention output vectors for A, B, C:")
print(output)
This code prints the new, context-enriched vector for each word after self-attention. Each result blends information from every word and shows how the output shifts based on what each word attended to.
Self-attention lets every word immediately access and weigh all other words in the sequence. It doesn’t need to pass memory along step by step, like RNNs. It doesn’t have to rely on local windows, like CNNs.
This has big effects:
These qualities made self-attention—especially as used in the Transformer model—a foundation for modern language and sequence models. It enables models to capture relationships in data more flexibly and effectively than previous approaches.
Try implementing a simple self-attention calculation for three word embeddings: [1,0], [0,1], and [1,1]. Assign each as the query, key, and value for the corresponding word. For each word, compute dot products with all keys to get similarity scores, apply softmax to these scores, and then use the resulting weights to take a weighted average of the value vectors. Do this by hand or in code, and compare the computed outputs for each word.
Coming up on Day 28: Multi-Head Attention