{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Priyesh Dave",
  "title_listed": "Day 11: Why One-Hot Encoding Fails: The Case for Embeddings",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/priyeshdave6/day-11-why-one-hot-encoding-fails-the-case-for-embeddings-2037",
  "publish_date": "2026-09-04",
  "corpus_stage": "",
  "notes": "",
  "id": "82d51f6d2112",
  "collected_at": "2026-10-02T13:50:02",
  "title": "Day 11: Why One-Hot Encoding Fails: The Case for Embeddings - DEV Community",
  "body_markdown": "What is One-Hot Encoding?\nComputers need numbers, not words. To process language, we first turn words into numbers. One of the earliest ways to do this is\none-hot encoding\n.\nA one-hot encoding represents each word in your vocabulary with a vector—a list of numbers—whose length matches the vocabulary size. All entries in this list are zero, except for a single one: the “hot” bit, which marks the position of that word in your list.\nThis approach is simple to compute. It lets you plug words into early machine learning algorithms. One-hot encoding was a workhorse for the first language models.\nExample: One-Hot Vectors\nSuppose your vocabulary is just three animals:\ncat\n,\ndog\n, and\nfish\n. Assign each a position:\ncat\nis at position 0\ndog\nis at position 1\nfish\nis at position 2\nTheir one-hot encodings are:\ncat\n→ [1, 0, 0]\ndog\n→ [0, 1, 0]\nfish\n→ [0, 0, 1]\nIf you see\n[0, 1, 0]\n, you know it’s\ndog\nbecause only the middle spot is “hot.”\nDrawbacks of One-Hot Encoding\nNow scale up: your vocabulary has tens of thousands of words. Each word becomes a long vector with mostly zeros and a single one.\nThis causes several issues:\n1. No Sense of Similarity\nOne-hot vectors don’t reflect meaning or relationships. For the computer,\ncat\nand\ndog\nlook just as unrelated as\ncat\nand\nfish\n. There’s no sense that dogs and cats are both pets or mammals.\nThe\ndot product\n—a way to measure how much two vectors point in the same direction—is zero for any pair of different one-hot vectors.\nThe\ndistance\nbetween any two distinct one-hot vectors is always the same.\nExample in code:\nimport\nnumpy\nas\nnp\n# Vocabulary and lookup\nvocab\n=\n[\n'\ncat\n'\n,\n'\ndog\n'\n,\n'\nfish\n'\n]\nword_to_index\n=\n{\nword\n:\nidx\nfor\nidx\n,\nword\nin\nenumerate\n(\nvocab\n)}\n# Function to create a one-hot vector\ndef\none_hot\n(\nword\n):\nvec\n=\nnp\n.\nzeros\n(\nlen\n(\nvocab\n))\nvec\n[\nword_to_index\n[\nword\n]]\n=\n1\nreturn\nvec\ncat_vec\n=\none_hot\n(\n'\ncat\n'\n)\ndog_vec\n=\none_hot\n(\n'\ndog\n'\n)\nfish_vec\n=\none_hot\n(\n'\nfish\n'\n)\n# Dot products\nprint\n(\nnp\n.\ndot\n(\ncat_vec\n,\ndog_vec\n))\n# 0.0\nprint\n(\nnp\n.\ndot\n(\ncat_vec\n,\nfish_vec\n))\n# 0.0\n# Euclidean distances\nprint\n(\nnp\n.\nlinalg\n.\nnorm\n(\ncat_vec\n-\ndog_vec\n))\n# 1.414...\nprint\n(\nnp\n.\nlinalg\n.\nnorm\n(\ncat_vec\n-\nfish_vec\n))\n# 1.414...\nEnter fullscreen mode\nExit fullscreen mode\nNo matter which two words you compare, the numbers are identical. One-hot vectors can't show that some words are more related than others.\n2. Inefficient Representation\nReal vocabularies are large. Each one-hot vector is mostly zeros—these are called\nsparse vectors\n. If you have 10,000 words, each word uses a vector 10,000 entries long. Storing these wastes memory and computation.\nWhy This Matters\nModern language tasks depend on recognizing relationships between words. Search engines, translation systems, and sentiment analysis systems need to recognize that words like\nhappy\nand\njoyful\nare connected in meaning.\nWith one-hot encoding, swapping a word for a synonym changes the vector completely. The computer sees no connection. This makes it hard for models to learn about patterns, context, or subtle meanings.\nIf a system can’t detect that two words mean nearly the same thing, it will struggle to find similar documents, suggest synonyms, or identify related concepts.\nA Glimpse at Embeddings\nTo address these problems, we use\nembeddings\n.\nEmbeddings also represent words as vectors. But instead of being all zeros except for one, embeddings use vectors filled with real values—often dozens or hundreds of numbers per word. Words with similar meanings get vectors that are close together.\nCat\nand\ndog\nembeddings are similar;\ncat\nand\nfish\nare not as close.\nThis lets computers detect patterns, similarity, and context. You’ll learn how embeddings work, and how they power modern language models, in upcoming lessons. For now: embeddings let machines measure similarity and capture relationships between words, something one-hot encoding can’t do.\nKey Takeaways\nOne-hot encoding maps each word to a unique sparse vector with a single one.\nAll one-hot vectors are equally distant; they show no word similarity.\nOne-hot representations become inefficient with large vocabularies.\nEmbeddings create dense vectors that capture relationships and similarity between words.\nTry It Yourself\nChoose five words and assign each a position in your list. Write their one-hot vector representations either by hand or in code. Compare the vectors for each word pair—are any pairs more similar than others? Explain your findings.\nFurther Resources\n🎥\nWhat Are Word Embeddings?\n🎥\nwhy dense embeddings outshine one‑hot vector encoding: Intuitive explanation\n📄\nEmbeddings | Transformer 101\n📄\nEmbedding Layer | Neel Mishra\n📄\nWord Representations and the Foundations of Word Embeddings\nComing up on Day 12:\nWord2Vec Explained: Skip-Gram and CBOW",
  "metadata": {
    "title": "Day 11: Why One-Hot Encoding Fails: The Case for Embeddings - DEV Community",
    "description": "What is One-Hot Encoding?  Computers need numbers, not words. To process language, we first turn... Tagged with ai, nlp, genai, 100daysofcode.",
    "keywords": "ai, nlp, genai, 100daysofcode, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/priyeshdave6/day-11-why-one-hot-encoding-fails-the-case-for-embeddings-2037",
    "og:title": "Day 11: Why One-Hot Encoding Fails: The Case for Embeddings",
    "og:description": "What is One-Hot Encoding?  Computers need numbers, not words. To process language, we first turn...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F05rxqlzt1hj5idn1mat3.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Day 11: Why One-Hot Encoding Fails: The Case for Embeddings",
    "twitter:description": "What is One-Hot Encoding?  Computers need numbers, not words. To process language, we first turn...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F05rxqlzt1hj5idn1mat3.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/82d51f6d2112/code/body-000.txt",
      "chars": 579,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=hVM8qGRTaOA",
      "src": "https://www.youtube.com/watch?v=hVM8qGRTaOA",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/82d51f6d2112/videos/repair-hVM8qGRTaOA.mp4",
      "bytes": 38633164,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "1172.101000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=chF3BoGeSG8",
      "src": "https://www.youtube.com/watch?v=chF3BoGeSG8",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/82d51f6d2112/videos/repair-chF3BoGeSG8.mp4",
      "bytes": 64188391,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "241.881000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://www.youtube.com/watch?v=hVM8qGRTaOA",
      "label": "What Are Word Embeddings?",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=chF3BoGeSG8",
      "label": "why dense embeddings outshine one‑hot vector encoding: Intuitive explanation",
      "type": "a"
    },
    {
      "href": "https://www.transformer101.com/embeddings",
      "label": "Embeddings | Transformer 101",
      "type": "a"
    },
    {
      "href": "https://neelmishra.github.io/blog/dl/transformers/building-blocks/embedding-layer.html",
      "label": "Embedding Layer | Neel Mishra",
      "type": "a"
    },
    {
      "href": "https://insightful-data-lab.com/2025/07/15/word-representations-and-the-foundations-of-word-embeddings/",
      "label": "Word Representations and the Foundations of Word Embeddings",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:11Z",
  "source_url": "https://dev.to/priyeshdave6/day-11-why-one-hot-encoding-fails-the-case-for-embeddings-2037",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 4777,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/82d51f6d2112/pages/source.html",
  "local_html": "attachments/82d51f6d2112/pages/article.html",
  "body_base_path": "attachments/82d51f6d2112/pages/",
  "body_markdown_path": "attachments/82d51f6d2112/pages/article.md"
}