{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Priyesh Dave",
  "title_listed": "Day 12: Word2Vec Explained: Skip-Gram and CBOW",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/priyeshdave6/day-12-word2vec-explained-skip-gram-and-cbow-31j5",
  "publish_date": "2026-09-06",
  "corpus_stage": "",
  "notes": "",
  "id": "9a2081e1654c",
  "collected_at": "2026-10-02T13:50:00",
  "title": "Day 12: Word2Vec Explained: Skip-Gram and CBOW - DEV Community",
  "body_markdown": "Previously, on Day 11:\nExplained one-hot encoding as a method for representing words as sparse vectors, highlighted its limitations regarding similarity and efficiency, and introduced the concept of word embeddings as an improvement.\nWhat Are Word2Vec Models?\nOne-hot vectors represent words for a computer using long lists of zeros with a single one—like a light switch for each word. Each word has a unique position. Every other position is zero. This approach is simple but has two big drawbacks. First, the vectors are extremely sparse—almost all zeros. Second, they contain no information about meaning. The one-hot vectors for “cat” and “dog” are just as different as those for “cat” and “banana,” even though “cat” and “dog” are related.\nWord2Vec\nis a technique that addresses both problems. It learns short, dense vectors called\nembeddings\n. Each word gets its own embedding—a list of a few dozen or a few hundred decimal numbers. These embeddings are not hand-written. They are trained from large amounts of real text. Words with similar meanings end up with similar embeddings.\nThe Skip-Gram Model: Predicting Context from a Target Word\nThe\nSkip-Gram\nmodel is one of two main approaches in Word2Vec. Its idea:\nGiven a single target word, can we guess which words are likely to appear near it in sentences?\nExample:\nTake the sentence:\n\"The quick brown fox jumps over the lazy dog\"\nEnter fullscreen mode\nExit fullscreen mode\nPick the word “fox.” Imagine a window that covers two words to the left and two to the right. The context words are: “quick,” “brown,” “jumps,” “over.”\nThe skip-gram model’s task is to take “fox” and try to predict each of those context words.\nDuring training, the model is shown pairs like (“fox”, “quick”), (“fox”, “brown”), and so on. But it doesn’t just memorize the examples. Each word is mapped to a vector, and the model tweaks these vectors so the right target-context pairs become more likely, over millions of examples.\nOver time, words like “fox,” “dog,” and “wolf” often show up in similar contexts. Their embeddings shift closer together in the vector space.\nThe CBOW Model: Predicting a Target Word from Its Context\nContinuous Bag of Words (CBOW)\nis the reverse of skip-gram. Instead of predicting context from a word, it predicts a single missing word given its surroundings.\nExample:\nConsider the same sentence:\n\"The quick brown fox jumps over the lazy dog\"\nEnter fullscreen mode\nExit fullscreen mode\nBlank out the word “fox.” Now the model sees: “quick,” “brown,” “jumps,” “over.” Its job is to predict that the missing word is “fox.”\nCBOW combines the embeddings for all the context words (by averaging or adding them) and uses that combined vector to predict the most likely target word. During training, it adjusts all the embeddings so that, given surrounding words, the correct word is predicted as often as possible.\nBoth skip-gram and CBOW learn useful word embeddings as a side effect of trying to solve their prediction tasks.\nComparing Skip-Gram and CBOW with Concrete Examples\nSuppose we use the sentence:\n\"The quick brown fox jumps\"\nEnter fullscreen mode\nExit fullscreen mode\nwith a context window of 2.\nSkip-Gram examples:\nInput: “brown” → Output: “quick”, “fox”\nInput: “fox” → Output: “brown”, “jumps”\nYou feed the model each word in the middle and try to guess nearby words.\nCBOW examples:\nInput: “quick”, “fox” → Output: “brown”\nInput: “brown”, “jumps” → Output: “fox”\nYou use the neighboring words as input and try to guess the middle word.\nKey points:\nSkip-gram generates more training pairs (since every word predicts several context words), but each pair is simple—one input, one output.\nCBOW creates fewer pairs, but each combines multiple input words into a single example.\nHow Training Adjusts Word Embeddings\nWhen the model predicts incorrectly—for example, guessing “wolf” instead of “fox”—it slightly changes the embeddings to reduce the error. If “quick,” “brown,” and “jumps” often occur near both “fox” and “wolf,” the embeddings for “fox” and “wolf” will become more similar.\nWords that appear in many environments, like “the” or “and,” get embeddings that reflect their generic context.\nWord2Vec training is entirely driven by the patterns in real text. Nothing about meaning is written in by hand. The frequencies and co-occurrences in the data teach the model what words often appear together. The resulting math places similar-meaning words near each other in the embedding space.\nOver many examples, clusters emerge. “King” and “queen” end up close together. “Walk” and “run” are neighbors. Sometimes, the difference between vectors is also meaningful: the vector from “man” to “woman” is similar to the vector from “king” to “queen.”\nWhen to Use Skip-Gram or CBOW\nBoth models learn good word vectors, but each has specific strengths.\nSkip-Gram:\nStronger for rare words. Every rare word gets a chance to predict its context in each appearance, so the model can learn their nuances even from limited data.\nCBOW:\nFaster to train. It predicts a word from several context words combined into one example, making it more efficient. It works well for common words but can blur details for rare or unusual words.\nChoose skip-gram if your data contains many unique or rare words and you care about precise representations. Choose CBOW if you have huge amounts of text and want fast, general embeddings.\nCode Example: Minimal Skip-Gram with Gensim\nHere’s a simple example using\ngensim\n, a Python library for Word2Vec. This code trains a skip-gram model on a tiny dataset and finds the most similar words to “fox.”\nfrom\ngensim.models\nimport\nWord2Vec\n# Small demo corpus\nsentences\n=\n[\n[\n\"\nthe\n\"\n,\n\"\nquick\n\"\n,\n\"\nbrown\n\"\n,\n\"\nfox\n\"\n,\n\"\njumps\n\"\n],\n[\n\"\nthe\n\"\n,\n\"\nlazy\n\"\n,\n\"\ndog\n\"\n,\n\"\nsleeps\n\"\n],\n[\n\"\nthe\n\"\n,\n\"\nfox\n\"\n,\n\"\noutsmarts\n\"\n,\n\"\nthe\n\"\n,\n\"\ndog\n\"\n],\n]\n# Train a skip-gram Word2Vec model\nmodel\n=\nWord2Vec\n(\nsentences\n,\n# list of tokenized sentences\nvector_size\n=\n50\n,\n# embedding dimensions\nwindow\n=\n2\n,\n# context window size\nmin_count\n=\n1\n,\n# include all words\nsg\n=\n1\n,\n# 1 for skip-gram; 0 for CBOW\n)\n# Find most similar words to “fox”\nsimilar_words\n=\nmodel\n.\nwv\n.\nmost_similar\n(\n\"\nfox\n\"\n,\ntopn\n=\n3\n)\nfor\nword\n,\nscore\nin\nsimilar_words\n:\nprint\n(\nword\n,\nscore\n)\nEnter fullscreen mode\nExit fullscreen mode\nIn this snippet:\nEach sentence is tokenized as a list of words.\nsg=1\nchooses skip-gram. Switch to\nsg=0\nfor CBOW.\nAfter training, we ask for the three words closest to “fox.”\nWith such a tiny dataset, results are limited, but you’ll see “dog” or \"brown\" may appear—showing which embeddings are close even from a small set.\nWord2Vec can easily scale to millions of words and much larger texts. The process for training and looking up similar words is just the same. With enough data, the model finds meaningful relationships between words automatically.\nKey Takeaways\nOne-hot vectors lack meaning and are sparse, while Word2Vec embeddings are dense and semantic.\nSkip-gram predicts context words from a target word, generating multiple simple training pairs.\nCBOW predicts a missing word from its context, combining surrounding words in each example.\nSkip-gram excels with rare words; CBOW trains faster but can blur rare word distinctions.\nWord2Vec learns word meanings entirely from text data using co-occurrence patterns.\nTry It Yourself\nTake the sentence 'The cat sat on the mat.' For a context window of 1, write out all the Skip-Gram training pairs by hand: for each word, list the words directly before and after it as outputs. This will help you see how Skip-Gram training data is structured.\nFurther Resources\n🎥\nWord2Vec Explained - CBOW & Skip‑Gram Models (YouTube)\n📘\nTensorFlow Word2Vec Tutorial\n📄\nWord2Vec Demystified: From One‑Hot to Vector Magic (CBOW & Skip‑Gram Explained)\n📄\nword2vec Parameter Learning Explained (Xin Rong)\nComing up on Day 13:\nGloVe: Global Vectors for Word Representation",
  "metadata": {
    "title": "Day 12: Word2Vec Explained: Skip-Gram and CBOW - DEV Community",
    "description": "Previously, on Day 11: Explained one-hot encoding as a method for representing words as sparse... Tagged with ai, nlp, genai, 100daysofcode.",
    "keywords": "ai, nlp, genai, 100daysofcode, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/priyeshdave6/day-12-word2vec-explained-skip-gram-and-cbow-31j5",
    "og:title": "Day 12: Word2Vec Explained: Skip-Gram and CBOW",
    "og:description": "Previously, on Day 11: Explained one-hot encoding as a method for representing words as sparse...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7d6lcu00q5p09fem0b8s.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Day 12: Word2Vec Explained: Skip-Gram and CBOW",
    "twitter:description": "Previously, on Day 11: Explained one-hot encoding as a method for representing words as sparse...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7d6lcu00q5p09fem0b8s.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/9a2081e1654c/code/body-000.txt",
      "chars": 46,
      "language": "text"
    },
    {
      "file": "attachments/9a2081e1654c/code/body-001.txt",
      "chars": 46,
      "language": "text"
    },
    {
      "file": "attachments/9a2081e1654c/code/body-002.txt",
      "chars": 28,
      "language": "text"
    },
    {
      "file": "attachments/9a2081e1654c/code/body-003.txt",
      "chars": 636,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=QYrhJUBWJwA",
      "status": "ok",
      "local_path": "attachments/9a2081e1654c/videos/QYrhJUBWJwA.mp4",
      "bytes": 22577123,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "599.861000"
    }
  ],
  "links": [
    {
      "href": "https://www.youtube.com/watch?v=QYrhJUBWJwA",
      "label": "Word2Vec Explained - CBOW & Skip‑Gram Models (YouTube)",
      "type": "a"
    },
    {
      "href": "https://www.tensorflow.org/text/tutorials/word2vec",
      "label": "TensorFlow Word2Vec Tutorial",
      "type": "a"
    },
    {
      "href": "https://blog.subhampanda.com/word2vec-demystified-from-one-hot-to-vector-magic-cbow-skip-gram-explained",
      "label": "Word2Vec Demystified: From One‑Hot to Vector Magic (CBOW & Skip‑Gram Explained)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1411.2738",
      "label": "word2vec Parameter Learning Explained (Xin Rong)",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:10Z",
  "source_url": "https://dev.to/priyeshdave6/day-12-word2vec-explained-skip-gram-and-cbow-31j5",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1411.2738",
      "bytes": 816833,
      "reused": false,
      "local_path": "papers/repair/cccb2abe70c590433dc7.pdf"
    }
  ],
  "actual_video_files": [
    {
      "local_path": "attachments/9a2081e1654c/videos/QYrhJUBWJwA.mp4",
      "bytes": 22577123,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "599.861000",
      "status": "ok"
    }
  ],
  "body_chars": 7882,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/9a2081e1654c/pages/source.html",
  "local_html": "attachments/9a2081e1654c/pages/article.html",
  "body_base_path": "attachments/9a2081e1654c/pages/",
  "body_markdown_path": "attachments/9a2081e1654c/pages/article.md"
}