{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Priyesh Dave",
  "title_listed": "Day 13: GloVe: Global Vectors for Word Representation",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/priyeshdave6/day-13-glove-global-vectors-for-word-representation-1ng9",
  "publish_date": "2026-09-09",
  "corpus_stage": "",
  "notes": "",
  "id": "72f919ad0a05",
  "collected_at": "2026-10-02T13:49:57",
  "title": "Day 13: GloVe: Global Vectors for Word Representation - DEV Community",
  "body_markdown": "Previously, on Day 12:\nExplained how Word2Vec models generate word embeddings using skip-gram and CBOW techniques, compared their methods and trade-offs, and introduced a minimal example using Gensim.\nWhy GloVe? The Motivation Behind Global Word Embeddings\nWord2Vec is a powerful method for learning word vectors. It trains these vectors so that words appearing in similar\nlocal contexts\n—words that show up near the same neighbors—end up close together in the vector space. The model does this by predicting a word from its neighbors (the CBOW method) or by predicting a word’s neighbors from itself (the Skip-gram method). All training happens inside narrow “windows” of text.\nFocusing only on local context leaves gaps. If two words never appear close together in those windows but have similar broad patterns across the corpus, Word2Vec might miss that relationship.\nGloVe (Global Vectors for Word Representation) addresses this limitation. GloVe captures both local context and\nglobal statistics\n—patterns from the entire dataset. Instead of just “Who is next to ‘king’ in a sentence?”, GloVe asks: “Across everything, how often do ‘king’ and ‘queen’ co-occur with the same words? How does this compare to ‘king’ and ‘man’?” GloVe uses information from all word pairs across the dataset, not just small windows, to learn these connections.\nThe main insight is that\nhow often\ntwo words appear together, compared to how often they appear apart, reveals word meaning. For example, if “ice” appears with “cold” much more than “steam” does—and “steam” with “hot” much more than “ice”—those ratios reveal important facts about the words.\nHow GloVe Works: Counting Co-occurrences and Building Vectors\nGloVe builds on counting global word co-occurrences. Here’s the basic process:\nBuild a vocabulary:\nFor example, pick the top 400,000 most frequent words in your data.\nCount co-occurrences:\nFor every word pair (\ni\n,\nj\n), count how many times they appear near each other inside a window (say, five words on each side). This creates a\nco-occurrence matrix\n.\nExample: If \"cat\" appears within five words of \"pet\" 350 times, then the cell (\"cat\", \"pet\") in the matrix contains 350.\nCompute statistics:\nFor each word pair, you know: “Given word\ni\n, how likely is word\nj\nnearby?” You can turn this into a frequency, probability, or ratio.\nLearn word vectors:\nTrain each word’s vector so relationships between vectors match these statistics (details in the next section).\nThink of the co-occurrence matrix as an enormous spreadsheet. Each row and column is a word; each cell tells you how many times those words appear near each other anywhere in your data. (For efficiency, GloVe usually works just with the nonzero counts.)\nThe GloVe Objective: What the Model Learns\nWhat does GloVe actually\noptimize\n? The model tries to make the dot product of two word vectors (plus bias terms) match the logarithm of their co-occurrence count. In other words, the geometry of the vectors should reflect the global counts from your data.\nHere’s the simplified loss function:\n[\nJ = \\sum_{i,j=1}^{V} f(X_{ij})\\, \\left(w_i^T \\cdot w_j + b_i + b_j - \\log(X_{ij})\\right)^2\n]\nTerm by term:\nX_{ij}\n: How often word\nj\nappears near word\ni\nin the corpus.\nw_i\nand\nw_j\n: Vectors for words\ni\nand\nj\n.\nb_i\nand\nb_j\n: Bias values for each word—just single numbers.\nf(X_{ij})\n: A weighting function so rare or extremely frequent pairs don’t overly affect the loss.\nFor each pair of words, the model pushes their dot product (plus biases) to be as close as possible to the log of their co-occurrence count.\nConcrete Example:\nIf \"ice\" and \"cold\" appear together 500 times (\nX_{ice,cold}=500\n) and \"ice\" and \"steam\" only 12 times, then\nlog(500)\nis about 6.2 and\nlog(12)\nabout 2.5.\nGloVe trains the vectors for \"ice\" and \"cold\" so their dot product (+ biases) is close to 6.2, and \"ice\" + \"steam\" near 2.5.\nIf the model achieves this match for all word pairs, the words’ positions in vector space encode much of the useful statistical information in the corpus.\nWhy use logs and not raw counts? Logarithms compress a huge range of counts into more manageable values and make optimization easier.\nGloVe Embeddings in Practice: Downloading and Using Pretrained Vectors\nMost people use\npretrained\nGloVe vectors. These are available for English (and some other languages) from datasets like Wikipedia, Common Crawl (large web scrape), and news sources.\nA typical GloVe embedding file is a text file. Each line contains a word and its vector of numbers:\nword_1 0.418 0.24968 ... -0.68568\nword_2 0.123 -0.289 ... 1.459\n...\nEnter fullscreen mode\nExit fullscreen mode\nPretrained GloVe sets most commonly come in 50, 100, 200, or 300 dimensions. Vocabularies are very large—up to 400,000 words. You rarely need all of them; load only what you need for your project. The file format is simple and works almost interchangeably with Word2Vec’s text format.\nWhen are pretrained GloVe files useful?\nWhen you need word vectors quickly without heavy computation.\nWhen working in restricted data environments (medical, legal, or proprietary data), you can fine-tune these vectors on your domain, instead of training from scratch.\nQuickstart: Loading and Using GloVe Embeddings in Python\nBelow is a simple script to download a small GloVe file, load the vectors, and compute the similarity between two words.\nimport\nnumpy\nas\nnp\nimport\nurllib.request\nimport\nzipfile\nimport\nos\n# Download a small GloVe embedding set (50 dimensions, ~70MB zipped)\nglove_url\n=\n\"\nhttp://nlp.stanford.edu/data/glove.6B.zip\n\"\nglove_zip_path\n=\n\"\nglove.6B.zip\n\"\nif\nnot\nos\n.\npath\n.\nexists\n(\nglove_zip_path\n):\nurllib\n.\nrequest\n.\nurlretrieve\n(\nglove_url\n,\nglove_zip_path\n)\n# Extract the smallest file (glove.6B.50d.txt)\nwith\nzipfile\n.\nZipFile\n(\nglove_zip_path\n,\n'\nr\n'\n)\nas\nz\n:\nif\nnot\nos\n.\npath\n.\nexists\n(\n\"\nglove.6B.50d.txt\n\"\n):\nz\n.\nextract\n(\n\"\nglove.6B.50d.txt\n\"\n)\n# Load GloVe vectors into a dictionary\nembeddings\n=\n{}\nwith\nopen\n(\n\"\nglove.6B.50d.txt\n\"\n,\nencoding\n=\n\"\nutf-8\n\"\n)\nas\nf\n:\nfor\nline\nin\nf\n:\nparts\n=\nline\n.\nstrip\n().\nsplit\n()\nword\n=\nparts\n[\n0\n]\nvec\n=\nnp\n.\narray\n(\nparts\n[\n1\n:],\ndtype\n=\nnp\n.\nfloat32\n)\nembeddings\n[\nword\n]\n=\nvec\n# Compute cosine similarity between two words\ndef\ncosine_similarity\n(\nx\n,\ny\n):\nreturn\nnp\n.\ndot\n(\nx\n,\ny\n)\n/\n(\nnp\n.\nlinalg\n.\nnorm\n(\nx\n)\n*\nnp\n.\nlinalg\n.\nnorm\n(\ny\n))\nword1\n,\nword2\n=\n\"\nking\n\"\n,\n\"\nqueen\n\"\nvec1\n=\nembeddings\n[\nword1\n]\nvec2\n=\nembeddings\n[\nword2\n]\nsim\n=\ncosine_similarity\n(\nvec1\n,\nvec2\n)\nprint\n(\nf\n\"\nCosine similarity between\n'\n{\nword1\n}\n'\nand\n'\n{\nword2\n}\n'\n:\n{\nsim\n:\n.\n3\nf\n}\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nThis script downloads 50-dimensional GloVe vectors, loads them into a Python dictionary keyed by word, and computes cosine similarity—a common measure of how similar two vectors are—between \"king\" and \"queen\".\nComparing GloVe and Word2Vec: When To Use Each\nHow they work:\nWord2Vec\nlearns word vectors using local text windows, optimizing to predict words from nearby words (or vice versa).\nGloVe\nconstructs a large matrix of global word co-occurrence counts, then learns vectors so relationships between them reflect these global patterns.\nBoth produce one fixed-length vector per word.\nPractical differences:\nGloVe tends to encode broader global relationships, since it “sees” the whole dataset at once.\nWord2Vec is faster to train on smaller datasets or when training must run continuously on incoming text.\nPretrained GloVe vectors often work well out of the box for analogy and similarity tasks, especially where less frequent but globally insightful connections matter.\nWhen to prefer each:\nUse\nGloVe\nwhen: You want to leverage richer global context, need high-quality pretrained vectors, or your task benefits from word relationships visible at the corpus level.\nUse\nWord2Vec\nwhen: You’re working with a smaller in-house dataset, want to adapt as new data arrives, or need easy updates and control over local window settings.\nIn practice, both methods generate useful embeddings for modern NLP projects. Many teams try both and keep whichever works best.\nKey Takeaways\nGloVe captures both local context and global co-occurrence statistics from the entire corpus.\nThe model optimizes word vectors so their dot products approximate the logarithm of word co-occurrence counts.\nPretrained GloVe embeddings can be easily used or fine-tuned for various NLP tasks.\nGloVe and Word2Vec differ in their approach: GloVe uses global statistics, Word2Vec focuses on local windows.\nGloVe vectors are often preferred when broad, corpus-level relationships are important.\nTry It Yourself\nDownload a different pretrained GloVe vector set, such as the Twitter embeddings. Load the vectors and write code to compute the five words most similar to 'king' using cosine similarity. Reflect on how these nearest neighbors compare to those produced by Word2Vec in a previous exercise.\nFurther Resources\n🎥\nLecture 3 | GloVe: Global Vectors for Word Representation (Stanford CS224n lecture)\n📘\nGloVe: Global Vectors for Word Representation (Stanford NLP project page)\nComing up on Day 14:\nFastText and Subword Embeddings",
  "metadata": {
    "title": "Day 13: GloVe: Global Vectors for Word Representation - DEV Community",
    "description": "Previously, on Day 12: Explained how Word2Vec models generate word embeddings using skip-gram and... Tagged with ai, nlp, genai, 100daysofcode.",
    "keywords": "ai, nlp, genai, 100daysofcode, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/priyeshdave6/day-13-glove-global-vectors-for-word-representation-1ng9",
    "og:title": "Day 13: GloVe: Global Vectors for Word Representation",
    "og:description": "Previously, on Day 12: Explained how Word2Vec models generate word embeddings using skip-gram and...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flwcf4ima0sx9z5n3aj8j.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Day 13: GloVe: Global Vectors for Word Representation",
    "twitter:description": "Previously, on Day 12: Explained how Word2Vec models generate word embeddings using skip-gram and...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flwcf4ima0sx9z5n3aj8j.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/72f919ad0a05/code/body-000.txt",
      "chars": 68,
      "language": "text"
    },
    {
      "file": "attachments/72f919ad0a05/code/body-001.txt",
      "chars": 1107,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=ASn7ExxLZws",
      "src": "https://www.youtube.com/watch?v=ASn7ExxLZws",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/72f919ad0a05/videos/repair-ASn7ExxLZws.mp4",
      "bytes": 238952473,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "4719.361000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://www.youtube.com/watch?v=ASn7ExxLZws",
      "label": "Lecture 3 | GloVe: Global Vectors for Word Representation (Stanford CS224n lecture)",
      "type": "a"
    },
    {
      "href": "https://nlp.stanford.edu/projects/glove/",
      "label": "GloVe: Global Vectors for Word Representation (Stanford NLP project page)",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:10Z",
  "source_url": "https://dev.to/priyeshdave6/day-13-glove-global-vectors-for-word-representation-1ng9",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 9075,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/72f919ad0a05/pages/source.html",
  "local_html": "attachments/72f919ad0a05/pages/article.html",
  "body_base_path": "attachments/72f919ad0a05/pages/",
  "body_markdown_path": "attachments/72f919ad0a05/pages/article.md"
}