{ "source_file": "0930", "source_platform": "研究机构", "author": "Dive into Deep Learning", "title_listed": "Word Embedding（章节）", "keywords": "词向量,word2vec,GloVe", "domain": "文本处理", "content_type": "教程/书籍", "is_paid_flag": "否", "url": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/word2vec.html", "publish_date": "持续更新", "corpus_stage": "文本处理", "notes": "词嵌入章节", "id": "969532c64a35", "collected_at": "2026-10-02T16:07:42", "title": "15.1. Word Embedding (word2vec) — Dive into Deep Learning 1.0.3 documentation", "body_markdown": "Table Of Contents\n15.1.\nWord Embedding (word2vec)\n¶\nColab [pytorch]\nOpen the notebook in Colab\nColab [mxnet]\nOpen the notebook in Colab\nColab [jax]\nOpen the notebook in Colab\nColab [tensorflow]\nOpen the notebook in Colab\nSageMaker Studio Lab\nOpen the notebook in SageMaker Studio Lab\nNatural language is a complex system used to express meanings. In this\nsystem, words are the basic unit of the meaning. As the name implies,\nword vectors\nare vectors used to represent words, and can also be\nconsidered as feature vectors or representations of words. The technique\nof mapping words to real vectors is called\nword embedding\n. In recent\nyears, word embedding has gradually become the basic knowledge of\nnatural language processing.\n15.1.1.\nOne-Hot Vectors Are a Bad Choice\n¶\nWe used one-hot vectors to represent words (characters are words) in\nSection 9.5\n. Suppose that the number of different words\nin the dictionary (the dictionary size) is\n\\(N\\)\n, and each word\ncorresponds to a different integer (index) from\n\\(0\\)\nto\n\\(N-1\\)\n. To obtain the one-hot vector representation for any word\nwith index\n\\(i\\)\n, we create a length-\n\\(N\\)\nvector with all 0s\nand set the element at position\n\\(i\\)\nto 1. In this way, each word\nis represented as a vector of length\n\\(N\\)\n, and it can be used\ndirectly by neural networks.\nAlthough one-hot word vectors are easy to construct, they are usually\nnot a good choice. A main reason is that one-hot word vectors cannot\naccurately express the similarity between different words, such as the\ncosine similarity\nthat we often use. For vectors\n\\(\\mathbf{x}, \\mathbf{y} \\in \\mathbb{R}^d\\)\n, their cosine similarity\nis the cosine of the angle between them:\n(15.1.1)\n¶\n\\[\\frac{\\mathbf{x}^\\top \\mathbf{y}}{\\|\\mathbf{x}\\| \\|\\mathbf{y}\\|} \\in [-1, 1].\\]\nSince the cosine similarity between one-hot vectors of any two different\nwords is 0, one-hot vectors cannot encode similarities among words.\n15.1.2.\nSelf-Supervised word2vec\n¶\nThe\nword2vec\ntool was\nproposed to address the above issue. It maps each word to a fixed-length\nvector, and these vectors can better express the similarity and analogy\nrelationship among different words. The word2vec tool contains two\nmodels, namely\nskip-gram\n(\nMikolov\net al.\n, 2013\n)\nand\ncontinuous bag of words\n(CBOW)\n(\nMikolov\net al.\n, 2013\n)\n.\nFor semantically meaningful representations, their training relies on\nconditional probabilities that can be viewed as predicting some words\nusing some of their surrounding words in corpora. Since supervision\ncomes from the data without labels, both skip-gram and continuous bag of\nwords are self-supervised models.\nIn the following, we will introduce these two models and their training\nmethods.\n15.1.3.\nThe Skip-Gram Model\n¶\nThe\nskip-gram\nmodel assumes that a word can be used to generate its\nsurrounding words in a text sequence. Take the text sequence “the”,\n“man”, “loves”, “his”, “son” as an example. Let’s choose “loves” as the\ncenter word\nand set the context window size to 2. As shown in\nFig. 15.1.1\n, given the center word “loves”, the skip-gram\nmodel considers the conditional probability for generating the\ncontext\nwords\n: “the”, “man”, “his”, and “son”, which are no more than 2 words\naway from the center word:\n(15.1.2)\n¶\n\\[P(\\textrm{\"the\"},\\textrm{\"man\"},\\textrm{\"his\"},\\textrm{\"son\"}\\mid\\textrm{\"loves\"}).\\]\nAssume that the context words are independently generated given the\ncenter word (i.e., conditional independence). In this case, the above\nconditional probability can be rewritten as\n(15.1.3)\n¶\n\\[P(\\textrm{\"the\"}\\mid\\textrm{\"loves\"})\\cdot P(\\textrm{\"man\"}\\mid\\textrm{\"loves\"})\\cdot P(\\textrm{\"his\"}\\mid\\textrm{\"loves\"})\\cdot P(\\textrm{\"son\"}\\mid\\textrm{\"loves\"}).\\]\nFig. 15.1.1\nThe skip-gram model considers the conditional probability of\ngenerating the surrounding context words given a center word.\n¶\nIn the skip-gram model, each word has two\n\\(d\\)\n-dimensional-vector\nrepresentations for calculating conditional probabilities. More\nconcretely, for any word with index\n\\(i\\)\nin the dictionary, denote\nby\n\\(\\mathbf{v}_i\\in\\mathbb{R}^d\\)\nand\n\\(\\mathbf{u}_i\\in\\mathbb{R}^d\\)\nits two vectors when used as a\ncenter\nword and a\ncontext\nword, respectively. The conditional\nprobability of generating any context word\n\\(w_o\\)\n(with index\n\\(o\\)\nin the dictionary) given the center word\n\\(w_c\\)\n(with\nindex\n\\(c\\)\nin the dictionary) can be modeled by a softmax operation\non vector dot products:\n(15.1.4)\n¶\n\\[P(w_o \\mid w_c) = \\frac{\\exp(\\mathbf{u}_o^\\top \\mathbf{v}_c)}{ \\sum_{i \\in \\mathcal{V}} \\exp(\\mathbf{u}_i^\\top \\mathbf{v}_c)},\\]\nwhere the vocabulary index set\n\\(\\mathcal{V} = \\{0, 1, \\ldots, |\\mathcal{V}|-1\\}\\)\n. Given a text\nsequence of length\n\\(T\\)\n, where the word at time step\n\\(t\\)\nis\ndenoted as\n\\(w^{(t)}\\)\n. Assume that context words are independently\ngenerated given any center word. For context window size\n\\(m\\)\n, the\nlikelihood function of the skip-gram model is the probability of\ngenerating all context words given any center word:\n(15.1.5)\n¶\n\\[\\prod_{t=1}^{T} \\prod_{-m \\leq j \\leq m,\\ j \\neq 0} P(w^{(t+j)} \\mid w^{(t)}),\\]\nwhere any time step that is less than\n\\(1\\)\nor greater than\n\\(T\\)\ncan be omitted.\n15.1.3.1.\nTraining\n¶\nThe skip-gram model parameters are the center word vector and context\nword vector for each word in the vocabulary. In training, we learn the\nmodel parameters by maximizing the likelihood function (i.e., maximum\nlikelihood estimation). This is equivalent to minimizing the following\nloss function:\n(15.1.6)\n¶\n\\[- \\sum_{t=1}^{T} \\sum_{-m \\leq j \\leq m,\\ j \\neq 0} \\textrm{log}\\, P(w^{(t+j)} \\mid w^{(t)}).\\]\nWhen using stochastic gradient descent to minimize the loss, in each\niteration we can randomly sample a shorter subsequence to calculate the\n(stochastic) gradient for this subsequence to update the model\nparameters. To calculate this (stochastic) gradient, we need to obtain\nthe gradients of the log conditional probability with respect to the\ncenter word vector and the context word vector. In general, according to\n(15.1.4)\nthe log conditional probability\ninvolving any pair of the center word\n\\(w_c\\)\nand the context word\n\\(w_o\\)\nis\n(15.1.7)\n¶\n\\[\\log P(w_o \\mid w_c) =\\mathbf{u}_o^\\top \\mathbf{v}_c - \\log\\left(\\sum_{i \\in \\mathcal{V}} \\exp(\\mathbf{u}_i^\\top \\mathbf{v}_c)\\right).\\]\nThrough differentiation, we can obtain its gradient with respect to the\ncenter word vector\n\\(\\mathbf{v}_c\\)\nas\n(15.1.8)\n¶\n\\[\\begin{split}\\begin{aligned}\\frac{\\partial \\textrm{log}\\, P(w_o \\mid w_c)}{\\partial \\mathbf{v}_c}&= \\mathbf{u}_o - \\frac{\\sum_{j \\in \\mathcal{V}} \\exp(\\mathbf{u}_j^\\top \\mathbf{v}_c)\\mathbf{u}_j}{\\sum_{i \\in \\mathcal{V}} \\exp(\\mathbf{u}_i^\\top \\mathbf{v}_c)}\\\\&= \\mathbf{u}_o - \\sum_{j \\in \\mathcal{V}} \\left(\\frac{\\exp(\\mathbf{u}_j^\\top \\mathbf{v}_c)}{ \\sum_{i \\in \\mathcal{V}} \\exp(\\mathbf{u}_i^\\top \\mathbf{v}_c)}\\right) \\mathbf{u}_j\\\\&= \\mathbf{u}_o - \\sum_{j \\in \\mathcal{V}} P(w_j \\mid w_c) \\mathbf{u}_j.\\end{aligned}\\end{split}\\]\nNote that the calculation in\n(15.1.8)\nrequires the\nconditional probabilities of all words in the dictionary with\n\\(w_c\\)\nas the center word. The gradients for the other word vectors\ncan be obtained in the same way.\nAfter training, for any word with index\n\\(i\\)\nin the dictionary, we\nobtain both word vectors\n\\(\\mathbf{v}_i\\)\n(as the center word) and\n\\(\\mathbf{u}_i\\)\n(as the context word). In natural language\nprocessing applications, the center word vectors of the skip-gram model\nare typically used as the word representations.\n15.1.4.\nThe Continuous Bag of Words (CBOW) Model\n¶\nThe\ncontinuous bag of words\n(CBOW) model is similar to the skip-gram\nmodel. The major difference from the skip-gram model is that the\ncontinuous bag of words model assumes that a center word is generated\nbased on its surrounding context words in the text sequence. For\nexample, in the same text sequence “the”, “man”, “loves”, “his”, and\n“son”, with “loves” as the center word and the context window size being\n2, the continuous bag of words model considers the conditional\nprobability of generating the center word “loves” based on the context\nwords “the”, “man”, “his” and “son” (as shown in\nFig. 15.1.2\n),\nwhich is\n(15.1.9)\n¶\n\\[P(\\textrm{\"loves\"}\\mid\\textrm{\"the\"},\\textrm{\"man\"},\\textrm{\"his\"},\\textrm{\"son\"}).\\]\nFig. 15.1.2\nThe continuous bag of words model considers the conditional\nprobability of generating the center word given its surrounding\ncontext words.\n¶\nSince there are multiple context words in the continuous bag of words\nmodel, these context word vectors are averaged in the calculation of the\nconditional probability. Specifically, for any word with index\n\\(i\\)\nin the dictionary, denote by\n\\(\\mathbf{v}_i\\in\\mathbb{R}^d\\)\nand\n\\(\\mathbf{u}_i\\in\\mathbb{R}^d\\)\nits two vectors when used as a\ncontext\nword and a\ncenter\nword (meanings are switched in the\nskip-gram model), respectively. The conditional probability of\ngenerating any center word\n\\(w_c\\)\n(with index\n\\(c\\)\nin the\ndictionary) given its surrounding context words\n\\(w_{o_1}, \\ldots, w_{o_{2m}}\\)\n(with index\n\\(o_1, \\ldots, o_{2m}\\)\nin the dictionary) can be modeled by\n(15.1.10)\n¶\n\\[P(w_c \\mid w_{o_1}, \\ldots, w_{o_{2m}}) = \\frac{\\exp\\left(\\frac{1}{2m}\\mathbf{u}_c^\\top (\\mathbf{v}_{o_1} + \\ldots + \\mathbf{v}_{o_{2m}}) \\right)}{ \\sum_{i \\in \\mathcal{V}} \\exp\\left(\\frac{1}{2m}\\mathbf{u}_i^\\top (\\mathbf{v}_{o_1} + \\ldots + \\mathbf{v}_{o_{2m}}) \\right)}.\\]\nFor brevity, let\n\\(\\mathcal{W}_o= \\{w_{o_1}, \\ldots, w_{o_{2m}}\\}\\)\nand\n\\(\\bar{\\mathbf{v}}_o = \\left(\\mathbf{v}_{o_1} + \\ldots + \\mathbf{v}_{o_{2m}} \\right)/(2m)\\)\n.\nThen\n(15.1.10)\ncan be simplified as\n(15.1.11)\n¶\n\\[P(w_c \\mid \\mathcal{W}_o) = \\frac{\\exp\\left(\\mathbf{u}_c^\\top \\bar{\\mathbf{v}}_o\\right)}{\\sum_{i \\in \\mathcal{V}} \\exp\\left(\\mathbf{u}_i^\\top \\bar{\\mathbf{v}}_o\\right)}.\\]\nGiven a text sequence of length\n\\(T\\)\n, where the word at time step\n\\(t\\)\nis denoted as\n\\(w^{(t)}\\)\n. For context window size\n\\(m\\)\n, the likelihood function of the continuous bag of words model\nis the probability of generating all center words given their context\nwords:\n(15.1.12)\n¶\n\\[\\prod_{t=1}^{T} P(w^{(t)} \\mid w^{(t-m)}, \\ldots, w^{(t-1)}, w^{(t+1)}, \\ldots, w^{(t+m)}).\\]\n15.1.4.1.\nTraining\n¶\nTraining continuous bag of words models is almost the same as training\nskip-gram models. The maximum likelihood estimation of the continuous\nbag of words model is equivalent to minimizing the following loss\nfunction:\n(15.1.13)\n¶\n\\[-\\sum_{t=1}^T \\textrm{log}\\, P(w^{(t)} \\mid w^{(t-m)}, \\ldots, w^{(t-1)}, w^{(t+1)}, \\ldots, w^{(t+m)}).\\]\nNotice that\n(15.1.14)\n¶\n\\[\\log\\,P(w_c \\mid \\mathcal{W}_o) = \\mathbf{u}_c^\\top \\bar{\\mathbf{v}}_o - \\log\\,\\left(\\sum_{i \\in \\mathcal{V}} \\exp\\left(\\mathbf{u}_i^\\top \\bar{\\mathbf{v}}_o\\right)\\right).\\]\nThrough differentiation, we can obtain its gradient with respect to any\ncontext word vector\n\\(\\mathbf{v}_{o_i}\\)\n(\n\\(i = 1, \\ldots, 2m\\)\n) as\n(15.1.15)\n¶\n\\[\\frac{\\partial \\log\\, P(w_c \\mid \\mathcal{W}_o)}{\\partial \\mathbf{v}_{o_i}} = \\frac{1}{2m} \\left(\\mathbf{u}_c - \\sum_{j \\in \\mathcal{V}} \\frac{\\exp(\\mathbf{u}_j^\\top \\bar{\\mathbf{v}}_o)\\mathbf{u}_j}{ \\sum_{i \\in \\mathcal{V}} \\exp(\\mathbf{u}_i^\\top \\bar{\\mathbf{v}}_o)} \\right) = \\frac{1}{2m}\\left(\\mathbf{u}_c - \\sum_{j \\in \\mathcal{V}} P(w_j \\mid \\mathcal{W}_o) \\mathbf{u}_j \\right).\\]\nThe gradients for the other word vectors can be obtained in the same\nway. Unlike the skip-gram model, the continuous bag of words model\ntypically uses context word vectors as the word representations.\n15.1.5.\nSummary\n¶\nWord vectors are vectors used to represent words, and can also be\nconsidered as feature vectors or representations of words. The\ntechnique of mapping words to real vectors is called word embedding.\nThe word2vec tool contains both the skip-gram and continuous bag of\nwords models.\nThe skip-gram model assumes that a word can be used to generate its\nsurrounding words in a text sequence; while the continuous bag of\nwords model assumes that a center word is generated based on its\nsurrounding context words.\n15.1.6.\nExercises\n¶\nWhat is the computational complexity for calculating each gradient?\nWhat could be the issue if the dictionary size is huge?\nSome fixed phrases in English consist of multiple words, such as “new\nyork”. How to train their word vectors? Hint: see Section 4 in the\nword2vec paper\n(\nMikolov\net al.\n, 2013\n)\n.\nLet’s reflect on the word2vec design by taking the skip-gram model as\nan example. What is the relationship between the dot product of two\nword vectors in the skip-gram model and the cosine similarity? For a\npair of words with similar semantics, why may the cosine similarity\nof their word vectors (trained by the skip-gram model) be high?\nDiscussions\nTable Of Contents\n15.1. Word Embedding (word2vec)\n15.1.1. One-Hot Vectors Are a Bad Choice\n15.1.2. Self-Supervised word2vec\n15.1.3. The Skip-Gram Model\n15.1.3.1. Training\n15.1.4. The Continuous Bag of Words (CBOW) Model\n15.1.4.1. Training\n15.1.5. Summary\n15.1.6. Exercises\nPrevious\n15. Natural Language Processing: Pretraining\nNext\n15.2. Approximate Training", "metadata": { "title": "15.1. Word Embedding (word2vec) — Dive into Deep Learning 1.0.3 documentation", "description": null, "keywords": null, "author": null }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "ok", "url": "https://www.d2l.ai/_static/logo-with-text.png", "bytes": 87175, "reused": false, "local_path": "attachments/969532c64a35/images/body-509f124ba61493e64845.png", "src": "https://www.d2l.ai/_static/logo-with-text.png", "alt": "Dive into Deep Learning", "downloaded": true }, { "status": "ok", "url": "https://www.d2l.ai/_images/skip-gram.svg", "bytes": 25587, "reused": false, "local_path": "attachments/969532c64a35/images/body-b2d0c247da6a63147399.svg", "src": "https://www.d2l.ai/_images/skip-gram.svg", "alt": "../_images/skip-gram.svg", "downloaded": true }, { "status": "ok", "url": "https://www.d2l.ai/_images/cbow.svg", "bytes": 26485, "reused": false, "local_path": "attachments/969532c64a35/images/body-4844c5b013755906327b.svg", "src": "https://www.d2l.ai/_images/cbow.svg", "alt": "../_images/cbow.svg", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://www.d2l.ai/index.html", "label": "", "type": "a" }, { "href": "https://colab.research.google.com/github/d2l-ai/d2l-pytorch-colab/blob/master/chapter_natural-language-processing-pretraining/word2vec.ipynb", "label": "Colab [pytorch]", "type": "a" }, { "href": "https://colab.research.google.com/github/d2l-ai/d2l-en-colab/blob/master/chapter_natural-language-processing-pretraining/word2vec.ipynb", "label": "Colab [mxnet]", "type": "a" }, { "href": "https://colab.research.google.com/github/d2l-ai/d2l-jax-colab/blob/master/chapter_natural-language-processing-pretraining/word2vec.ipynb", "label": "Colab [jax]", "type": "a" }, { "href": "https://colab.research.google.com/github/d2l-ai/d2l-tensorflow-colab/blob/master/chapter_natural-language-processing-pretraining/word2vec.ipynb", "label": "Colab [tensorflow]", "type": "a" }, { "href": "https://studiolab.sagemaker.aws/import/github/d2l-ai/d2l-pytorch-sagemaker-studio-lab/blob/main/GettingStarted-D2L.ipynb", "label": "SageMaker Studio Lab", "type": "a" }, { "href": "https://www.d2l.ai/chapter_recurrent-neural-networks/rnn-scratch.html", "label": "Section 9.5", "type": "a" }, { "href": "https://code.google.com/archive/p/word2vec/", "label": "word2vec", "type": "a" }, { "href": "https://www.d2l.ai/chapter_references/zreferences.html", "label": "Mikolov et al. , 2013", "type": "a" }, { "href": "https://discuss.d2l.ai/t/381", "label": "Discussions", "type": "a" }, { "href": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/index.html", "label": "Previous 15. Natural Language Processing: Pretraining", "type": "a" }, { "href": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/approx-training.html", "label": "Next 15.2. Approximate Training", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:53:49Z", "source_url": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/word2vec.html", "papers": [], "actual_video_files": [], "body_chars": 12962, "expected_body_images": 3, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/969532c64a35/pages/source.html", "local_html": "attachments/969532c64a35/pages/article.html", "body_base_path": "attachments/969532c64a35/pages/", "body_markdown_path": "attachments/969532c64a35/pages/article.md" }