Previously, on Day 12: Explained how Word2Vec models generate word embeddings using skip-gram and CBOW techniques, compared their methods and trade-offs, and introduced a minimal example using Gensim.
Word2Vec is a powerful method for learning word vectors. It trains these vectors so that words appearing in similar local contexts—words that show up near the same neighbors—end up close together in the vector space. The model does this by predicting a word from its neighbors (the CBOW method) or by predicting a word’s neighbors from itself (the Skip-gram method). All training happens inside narrow “windows” of text.
Focusing only on local context leaves gaps. If two words never appear close together in those windows but have similar broad patterns across the corpus, Word2Vec might miss that relationship.
GloVe (Global Vectors for Word Representation) addresses this limitation. GloVe captures both local context and global statistics—patterns from the entire dataset. Instead of just “Who is next to ‘king’ in a sentence?”, GloVe asks: “Across everything, how often do ‘king’ and ‘queen’ co-occur with the same words? How does this compare to ‘king’ and ‘man’?” GloVe uses information from all word pairs across the dataset, not just small windows, to learn these connections.
The main insight is that how often two words appear together, compared to how often they appear apart, reveals word meaning. For example, if “ice” appears with “cold” much more than “steam” does—and “steam” with “hot” much more than “ice”—those ratios reveal important facts about the words.
GloVe builds on counting global word co-occurrences. Here’s the basic process:
i, j), count how many times they appear near each other inside a window (say, five words on each side). This creates a co-occurrence matrix.
i, how likely is word j nearby?” You can turn this into a frequency, probability, or ratio.Think of the co-occurrence matrix as an enormous spreadsheet. Each row and column is a word; each cell tells you how many times those words appear near each other anywhere in your data. (For efficiency, GloVe usually works just with the nonzero counts.)
What does GloVe actually optimize? The model tries to make the dot product of two word vectors (plus bias terms) match the logarithm of their co-occurrence count. In other words, the geometry of the vectors should reflect the global counts from your data.
Here’s the simplified loss function:
[
J = \sum_{i,j=1}^{V} f(X_{ij})\, \left(w_i^T \cdot w_j + b_i + b_j - \log(X_{ij})\right)^2
]
Term by term:
X_{ij}: How often word j appears near word i in the corpus.w_i and w_j: Vectors for words i and j.b_i and b_j: Bias values for each word—just single numbers.f(X_{ij}): A weighting function so rare or extremely frequent pairs don’t overly affect the loss.For each pair of words, the model pushes their dot product (plus biases) to be as close as possible to the log of their co-occurrence count.
Concrete Example:
X_{ice,cold}=500) and "ice" and "steam" only 12 times, then log(500) is about 6.2 and log(12) about 2.5.Why use logs and not raw counts? Logarithms compress a huge range of counts into more manageable values and make optimization easier.
Most people use pretrained GloVe vectors. These are available for English (and some other languages) from datasets like Wikipedia, Common Crawl (large web scrape), and news sources.
A typical GloVe embedding file is a text file. Each line contains a word and its vector of numbers:
word_1 0.418 0.24968 ... -0.68568
word_2 0.123 -0.289 ... 1.459
...
Pretrained GloVe sets most commonly come in 50, 100, 200, or 300 dimensions. Vocabularies are very large—up to 400,000 words. You rarely need all of them; load only what you need for your project. The file format is simple and works almost interchangeably with Word2Vec’s text format.
When are pretrained GloVe files useful?
Below is a simple script to download a small GloVe file, load the vectors, and compute the similarity between two words.
import numpy as np
import urllib.request
import zipfile
import os
# Download a small GloVe embedding set (50 dimensions, ~70MB zipped)
glove_url = "http://nlp.stanford.edu/data/glove.6B.zip"
glove_zip_path = "glove.6B.zip"
if not os.path.exists(glove_zip_path):
urllib.request.urlretrieve(glove_url, glove_zip_path)
# Extract the smallest file (glove.6B.50d.txt)
with zipfile.ZipFile(glove_zip_path, 'r') as z:
if not os.path.exists("glove.6B.50d.txt"):
z.extract("glove.6B.50d.txt")
# Load GloVe vectors into a dictionary
embeddings = {}
with open("glove.6B.50d.txt", encoding="utf-8") as f:
for line in f:
parts = line.strip().split()
word = parts[0]
vec = np.array(parts[1:], dtype=np.float32)
embeddings[word] = vec
# Compute cosine similarity between two words
def cosine_similarity(x, y):
return np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y))
word1, word2 = "king", "queen"
vec1 = embeddings[word1]
vec2 = embeddings[word2]
sim = cosine_similarity(vec1, vec2)
print(f"Cosine similarity between '{word1}' and '{word2}': {sim:.3f}")
This script downloads 50-dimensional GloVe vectors, loads them into a Python dictionary keyed by word, and computes cosine similarity—a common measure of how similar two vectors are—between "king" and "queen".
How they work:
Practical differences:
When to prefer each:
In practice, both methods generate useful embeddings for modern NLP projects. Many teams try both and keep whichever works best.
Download a different pretrained GloVe vector set, such as the Twitter embeddings. Load the vectors and write code to compute the five words most similar to 'king' using cosine similarity. Reflect on how these nearest neighbors compare to those produced by Word2Vec in a previous exercise.
Coming up on Day 14: FastText and Subword Embeddings