Previously, on Day 11: Explained one-hot encoding as a method for representing words as sparse vectors, highlighted its limitations regarding similarity and efficiency, and introduced the concept of word embeddings as an improvement.
One-hot vectors represent words for a computer using long lists of zeros with a single one—like a light switch for each word. Each word has a unique position. Every other position is zero. This approach is simple but has two big drawbacks. First, the vectors are extremely sparse—almost all zeros. Second, they contain no information about meaning. The one-hot vectors for “cat” and “dog” are just as different as those for “cat” and “banana,” even though “cat” and “dog” are related.
Word2Vec is a technique that addresses both problems. It learns short, dense vectors called embeddings. Each word gets its own embedding—a list of a few dozen or a few hundred decimal numbers. These embeddings are not hand-written. They are trained from large amounts of real text. Words with similar meanings end up with similar embeddings.
The Skip-Gram model is one of two main approaches in Word2Vec. Its idea: Given a single target word, can we guess which words are likely to appear near it in sentences?
Example:
Take the sentence:
"The quick brown fox jumps over the lazy dog"
Pick the word “fox.” Imagine a window that covers two words to the left and two to the right. The context words are: “quick,” “brown,” “jumps,” “over.”
The skip-gram model’s task is to take “fox” and try to predict each of those context words.
During training, the model is shown pairs like (“fox”, “quick”), (“fox”, “brown”), and so on. But it doesn’t just memorize the examples. Each word is mapped to a vector, and the model tweaks these vectors so the right target-context pairs become more likely, over millions of examples.
Over time, words like “fox,” “dog,” and “wolf” often show up in similar contexts. Their embeddings shift closer together in the vector space.
Continuous Bag of Words (CBOW) is the reverse of skip-gram. Instead of predicting context from a word, it predicts a single missing word given its surroundings.
Example:
Consider the same sentence:
"The quick brown fox jumps over the lazy dog"
Blank out the word “fox.” Now the model sees: “quick,” “brown,” “jumps,” “over.” Its job is to predict that the missing word is “fox.”
CBOW combines the embeddings for all the context words (by averaging or adding them) and uses that combined vector to predict the most likely target word. During training, it adjusts all the embeddings so that, given surrounding words, the correct word is predicted as often as possible.
Both skip-gram and CBOW learn useful word embeddings as a side effect of trying to solve their prediction tasks.
Suppose we use the sentence:
"The quick brown fox jumps"
with a context window of 2.
Skip-Gram examples:
You feed the model each word in the middle and try to guess nearby words.
CBOW examples:
You use the neighboring words as input and try to guess the middle word.
Key points:
When the model predicts incorrectly—for example, guessing “wolf” instead of “fox”—it slightly changes the embeddings to reduce the error. If “quick,” “brown,” and “jumps” often occur near both “fox” and “wolf,” the embeddings for “fox” and “wolf” will become more similar.
Words that appear in many environments, like “the” or “and,” get embeddings that reflect their generic context.
Word2Vec training is entirely driven by the patterns in real text. Nothing about meaning is written in by hand. The frequencies and co-occurrences in the data teach the model what words often appear together. The resulting math places similar-meaning words near each other in the embedding space.
Over many examples, clusters emerge. “King” and “queen” end up close together. “Walk” and “run” are neighbors. Sometimes, the difference between vectors is also meaningful: the vector from “man” to “woman” is similar to the vector from “king” to “queen.”
Both models learn good word vectors, but each has specific strengths.
Choose skip-gram if your data contains many unique or rare words and you care about precise representations. Choose CBOW if you have huge amounts of text and want fast, general embeddings.
Here’s a simple example using gensim, a Python library for Word2Vec. This code trains a skip-gram model on a tiny dataset and finds the most similar words to “fox.”
from gensim.models import Word2Vec
# Small demo corpus
sentences = [
["the", "quick", "brown", "fox", "jumps"],
["the", "lazy", "dog", "sleeps"],
["the", "fox", "outsmarts", "the", "dog"],
]
# Train a skip-gram Word2Vec model
model = Word2Vec(
sentences, # list of tokenized sentences
vector_size=50, # embedding dimensions
window=2, # context window size
min_count=1, # include all words
sg=1, # 1 for skip-gram; 0 for CBOW
)
# Find most similar words to “fox”
similar_words = model.wv.most_similar("fox", topn=3)
for word, score in similar_words:
print(word, score)
In this snippet:
sg=1 chooses skip-gram. Switch to sg=0 for CBOW.Word2Vec can easily scale to millions of words and much larger texts. The process for training and looking up similar words is just the same. With enough data, the model finds meaningful relationships between words automatically.
Take the sentence 'The cat sat on the mat.' For a context window of 1, write out all the Skip-Gram training pairs by hand: for each word, list the words directly before and after it as outputs. This will help you see how Skip-Gram training data is structured.
Coming up on Day 13: GloVe: Global Vectors for Word Representation