{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "BAIR Blog (UC Berkeley)",
  "title_listed": "What exactly does word2vec learn?",
  "keywords": "word2vec,表示学习,理论",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://bair.berkeley.edu/blog/2025/09/01/qwem-word2vec-theory/",
  "publish_date": "2025-09",
  "corpus_stage": "预训练/SFT",
  "notes": "word2vec学习机制的定量理论",
  "id": "de937ea1a871",
  "collected_at": "2026-10-02T15:23:06",
  "title": "What exactly does word2vec learn? – The Berkeley Artificial Intelligence Research Blog",
  "body_markdown": "What exactly does\nword2vec\nlearn, and how? Answering this question amounts to understanding representation learning in a minimal yet interesting language modeling task. Despite the fact that\nword2vec\nis a well-known precursor to modern language models, for many years, researchers lacked a quantitative and predictive theory describing its learning process. In our new\npaper\n, we finally provide such a theory. We prove that there are realistic, practical regimes in which the learning problem reduces to\nunweighted least-squares matrix factorization\n. We solve the gradient flow dynamics in closed form; the final learned representations are simply given by PCA.\nLearning dynamics of word2vec\n. When trained from small initialization, word2vec learns in discrete, sequential steps. Left: rank-incrementing learning steps in the weight matrix, each decreasing the loss. Right: three time slices of the latent embedding space showing how embedding vectors expand into subspaces of increasing dimension at each learning step, continuing until model capacity is saturated.\nBefore elaborating on this result, let’s motivate the problem.\nword2vec\nis a well-known algorithm for learning dense vector representations of words. These embedding vectors are trained using a contrastive algorithm; at the end of training, the semantic relation between any two words is captured by the angle between the corresponding embeddings. In fact, the learned embeddings empirically exhibit striking linear structure in their geometry: linear subspaces in the latent space often encode interpretable concepts such as gender, verb tense, or dialect. This so-called\nlinear representation hypothesis\nhas recently garnered a lot of attention since\nLLMs exhibit this behavior as well\n, enabling\nsemantic inspection of internal representations\nand providing for\nnovel model steering techniques\n. In\nword2vec\n, it is precisely these linear directions that enable the learned embeddings to complete analogies (e.g., “man : woman :: king : queen”) via embedding vector addition.\nMaybe this shouldn’t be too surprising: after all, the\nword2vec\nalgorithm simply iterates through a text corpus and trains a two-layer linear network to model statistical regularities in natural language using self-supervised gradient descent. In this framing, it’s clear that\nword2vec\nis a minimal neural language model. Understanding\nword2vec\nis thus a prerequisite to understanding feature learning in more sophisticated language modeling tasks.\nThe Result\nWith this motivation in mind, let’s describe the main result. Concretely, suppose we initialize all the embedding vectors randomly and very close to the origin, so that they’re effectively zero-dimensional. Then (under some mild approximations) the embeddings collectively learn one “concept” (i.e., orthogonal linear subspace) at a time in a sequence of discrete learning steps.\nIt’s like when diving head-first into learning a new branch of math. At first, all the jargon is muddled — what’s the difference between a function and a functional? What about a linear operator vs. a matrix? Slowly, through exposure to new settings of interest, the words separate from each other in the mind and their true meanings become clearer.\nAs a consequence, each new realized linear concept effectively increments the rank of the embedding matrix, giving each word embedding more space to better express itself and its meaning. Since these linear subspaces do not rotate once they’re learned, these are effectively the model’s learned features. Our theory allows us to compute each of these features a priori in\nclosed form\n– they are simply the eigenvectors of a particular target matrix which is defined solely in terms of measurable corpus statistics and algorithmic hyperparameters.\nWhat are the features?\nThe answer is remarkably straightforward: the latent features are simply the top eigenvectors of the following matrix:\n\\[M^{\\star}_{ij} = \\frac{P(i,j) - P(i)P(j)}{\\frac{1}{2}(P(i,j) + P(i)P(j))}\\]\nwhere $i$ and $j$ index the words in the vocabulary, $P(i,j)$ is the co-occurrence probability for words $i$ and $j$, and $P(i)$ is the unigram probability for word $i$ (i.e., the marginal of $P(i,j)$).\nConstructing and diagonalizing this matrix from the Wikipedia statistics, one finds that the top eigenvector selects words associated with celebrity biographies, the second eigenvector selects words associated with government and municipal administration, the third is associated with geographical and cartographical descriptors, and so on.\nThe takeaway is this: during training,\nword2vec\nfinds a sequence of optimal low-rank approximations of $M^{\\star}$. It’s effectively equivalent to running PCA on $M^{\\star}$.\nThe following plots illustrate this behavior.\nLearning dynamics comparison showing discrete, sequential learning steps.\nOn the left, the key empirical observation is that\nword2vec\n(plus our mild approximations) learns in a sequence of essentially discrete steps. Each step increments the effective rank of the embeddings, resulting in a stepwise decrease in the loss. On the right, we show three time slices of the latent embedding space, demonstrating how the embeddings expand along a new orthogonal direction at each learning step. Furthermore, by inspecting the words that most strongly align with these singular directions, we observe that each discrete “piece of knowledge” corresponds to an interpretable topic-level concept. These learning dynamics are solvable in closed form, and we see an excellent match between the theory and numerical experiment.\nWhat are the mild approximations? They are: 1) quartic approximation of the objective function around the origin; 2) a particular constraint on the algorithmic hyperparameters; 3) sufficiently small initial embedding weights; and 4) vanishingly small gradient descent steps. Thankfully, these conditions are not too strong, and in fact they’re quite similar to the setting described in the original\nword2vec\npaper.\nImportantly, none of the approximations involve the data distribution! Indeed, a huge strength of the theory is that it makes no distributional assumptions. As a result, the theory predicts exactly what features are learned in terms of the corpus statistics and the algorithmic hyperparameters. This is particularly useful, since fine-grained descriptions of learning dynamics in the distribution-agnostic setting are rare and hard to obtain; to our knowledge, this is the first one for a practical natural language task.\nAs for the approximations we do make, we empirically show that our theoretical result still provides a faithful description of the original\nword2vec\n. As a coarse indicator of the agreement between our approximate setting and true\nword2vec\n, we can compare the empirical scores on the standard analogy completion benchmark:\nword2vec\nachieves 68% accuracy, the approximate model we study achieves 66%, and the standard classical alternative (known as PPMI) only gets 51%. Check out our paper to see plots with detailed comparisons.\nTo demonstrate the usefulness of the result, we apply our theory to study the emergence of abstract linear representations (corresponding to binary concepts such as masculine/feminine or past/future). We find that over the course of learning,\nword2vec\nbuilds these linear representations in a sequence of noisy learning steps, and their geometry is well-described by a spiked random matrix model. Early in training, semantic signal dominates; however, later in training, noise may begin to dominate, causing a degradation of the model’s ability to resolve the linear representation. See our paper for more details.\nAll in all, this result gives one of the first complete closed-form theories of feature learning in a minimal yet relevant natural language task. In this sense, we believe our work is an important step forward in the broader project of obtaining realistic analytical solutions describing the performance of practical machine learning algorithms.\nLearn more about our work:\nLink to full paper\nThis post originally appeared on\nDhruva Karkada’s blog\n.",
  "metadata": {
    "title": "What exactly does word2vec learn? – The Berkeley Artificial Intelligence Research Blog",
    "description": "The BAIR Blog",
    "keywords": "",
    "author": "C.K. Wolfe",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "What exactly does word2vec learn?",
    "og:description": "The BAIR Blog",
    "og:url": "http://bair.berkeley.edu/blog/2025/09/01/qwem-word2vec-theory/",
    "og:site_name": "The Berkeley Artificial Intelligence Research Blog",
    "og:image": "http://bair.berkeley.edu/blog/assets/qwem-word2vec-theory/fig1.c8u1a3E7_Z23iPso.webp",
    "og:image:url": "http://bair.berkeley.edu/blog/assets/qwem-word2vec-theory/fig1.c8u1a3E7_Z23iPso.webp"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://bair.berkeley.edu/static/blog/qwem-word2vec-theory/fig1.c8u1a3E7_Z23iPso.webp",
      "bytes": 60940,
      "reused": true,
      "local_path": "attachments/de937ea1a871/images/001.webp",
      "src": "https://bair.berkeley.edu/static/blog/qwem-word2vec-theory/fig1.c8u1a3E7_Z23iPso.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://bair.berkeley.edu/static/blog/qwem-word2vec-theory/fig2.C4kWlUSu_ZJTCeE.webp",
      "bytes": 62936,
      "reused": true,
      "local_path": "attachments/de937ea1a871/images/002.webp",
      "src": "https://bair.berkeley.edu/static/blog/qwem-word2vec-theory/fig2.C4kWlUSu_ZJTCeE.webp",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://arxiv.org/abs/2502.09863",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2311.03658",
      "label": "LLMs exhibit this behavior as well",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2309.00941",
      "label": "semantic inspection of internal representations",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2310.01405",
      "label": "novel model steering techniques",
      "type": "a"
    },
    {
      "href": "https://dkarkada.xyz/posts/qwem/",
      "label": "Dhruva Karkada’s blog",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:18Z",
  "source_url": "https://bair.berkeley.edu/blog/2025/09/01/qwem-word2vec-theory/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2502.09863",
      "bytes": 2038205,
      "reused": false,
      "local_path": "papers/repair/3c06bb9ecb06dfba2b14.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2311.03658",
      "bytes": 2534772,
      "reused": false,
      "local_path": "papers/repair/b883a95bca984fab69f7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2309.00941",
      "bytes": 3150501,
      "reused": false,
      "local_path": "papers/repair/3b97c45ed4ae37f07664.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2310.01405",
      "bytes": 10661161,
      "reused": false,
      "local_path": "papers/repair/f201d87cf0e942354c3a.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 8117,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/de937ea1a871/pages/source.html",
  "local_html": "attachments/de937ea1a871/pages/article.html",
  "body_base_path": "attachments/de937ea1a871/pages/",
  "body_markdown_path": "attachments/de937ea1a871/pages/article.md"
}