{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Priyesh Dave",
  "title_listed": "Day 14: FastText and Subword Embeddings",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/priyeshdave6/day-14-fasttext-and-subword-embeddings-34ed",
  "publish_date": "2026-09-09",
  "corpus_stage": "",
  "notes": "",
  "id": "af17b80f6cb0",
  "collected_at": "2026-10-02T13:49:58",
  "title": "Day 14: FastText and Subword Embeddings - DEV Community",
  "body_markdown": "What Are Word Embeddings?\nWords like \"cat,\" \"run,\" or \"happiness\" are just symbols to a computer. Computers operate on numbers, not symbols. To process language, we need a way to turn words into numbers—specifically, into mathematical objects that reflect their meaning and usage.\nWord embeddings solve this. An\nembedding\nis simply a list of numbers—a vector—that represents a word. If two words are used in similar contexts, their vectors should be close together. For example, \"cat\" and \"dog\" would have similar vectors; both are common animals, and often appear in similar situations.\nWord embeddings make it possible for machines to:\nCompare word meanings mathematically. (For example, using cosine similarity, which measures the angle between two vectors.)\nCluster similar words together in this new number-based space.\nUse these vectors as input features for machine learning models, including neural networks.\nBefore word embeddings, earlier approaches represented words in more primitive ways.\nThe Limits of Classic Embeddings\nThe most basic way to represent a word as a number is\none-hot encoding\n. In one-hot encoding, each word in the vocabulary gets its own vector. This vector contains a single \"1\" in the position for that word, and \"0\" everywhere else. If your vocabulary has 10,000 words, every word gets a 10,000-dimensional vector, with just one non-zero entry.\nThis approach has major problems:\nWastes memory. Almost every entry is zero.\nShows no relationships between words. \"Cat\" and \"dog\" are as far apart as \"cat\" and \"microwave.\"\nCannot handle words that were not seen during training time. These are called \"out-of-vocabulary\" (OOV) words.\nCannot handle typos. The model treats “cat” and “cta” as unrelated.\nStruggles with languages that have many forms of the same root word. Each form is treated as unrelated.\nWord embeddings such as Word2Vec or GloVe improved this by learning shorter, dense vectors for each word. Similar words have similar embeddings. But these classic embeddings still assign one unique vector per word. If a word is rare or was never seen in training, these models cannot create a vector for it.\nFastText: Embeddings With Subwords\nFastText, created by Facebook’s AI Research lab, addresses these gaps by looking inside words. Instead of treating a word as a single chunk, FastText represents each word as a bag of\nsubword units\n, also called\nn-grams\n.\nA subword unit is just a short piece of the word. For example, for the word “playing,” some 3-character n-grams are “pla,” “lay,” “ayi,” “yin,” and “ing.” FastText splits each word into overlapping character n-grams, usually between 3 and 6 characters long. Each n-gram gets its own vector.\nTo build the final embedding for a word, FastText combines the vectors for the word itself and all of its n-grams—typically by averaging or summing them.\nFor example, “jumping” would be represented by:\nThe embedding for the whole word “jumping”\nThe embeddings for “jum,” “ump,” “mpi,” “pin,” “ing” (and possibly others, depending on n-gram size)\nEarlier approaches had no understanding of parts within a word. FastText fixes this.\nWhy Subword Embeddings Matter\nSuppose the training data never included the word “jumped.” With classic embeddings, there is no vector for this word. The model cannot process it.\nWith FastText, \"jumped\" is split into n-grams such as \"jum,\" \"ump,\" \"mpe,\" \"ped.\" Even if “jumped” never showed up during training, its n-grams might have, from words like “jump,” “jumping,” or even “stumped.” FastText can build a vector for \"jumped\" by combining those parts.\nThis approach has important advantages:\nRobust to typos.\n“jumping” and a typo like “jum ping” share many n-grams, so their vectors are similar.\nHandles rare or invented words.\nIf a rare word like “quixotically” shares letter chunks with common words, it still gets a reasonable vector.\nWorks with morphologically rich languages.\nMany languages use lots of word endings to change meaning. FastText can represent all these forms, even if it never saw them in training.\nFastText is much less brittle than previous models, and handles natural, messy language well.\nWorking With FastText: A Minimal Example\nHere’s how to use FastText in practice. We’ll use Gensim, a Python library, to train a tiny FastText model. Then we’ll look up vectors for a common word, a rare word, and a totally new word.\nInstall Gensim:\npip\ninstall\ngensim\nEnter fullscreen mode\nExit fullscreen mode\nMinimal FastText code:\nfrom\ngensim.models\nimport\nFastText\n# Simple dataset: each sentence is a list of words.\ncorpus\n=\n[\n[\n\"\ncat\n\"\n,\n\"\nsat\n\"\n,\n\"\non\n\"\n,\n\"\nthe\n\"\n,\n\"\nmat\n\"\n],\n[\n\"\ndog\n\"\n,\n\"\nsat\n\"\n,\n\"\nbeside\n\"\n,\n\"\nthe\n\"\n,\n\"\ncat\n\"\n],\n[\n\"\ndogs\n\"\n,\n\"\nand\n\"\n,\n\"\ncats\n\"\n,\n\"\nare\n\"\n,\n\"\nfriends\n\"\n],\n[\n\"\nthe\n\"\n,\n\"\ndog\n\"\n,\n\"\njumped\n\"\n,\n\"\nover\n\"\n,\n\"\nthe\n\"\n,\n\"\nlog\n\"\n]\n]\n# Train a FastText model. Kept small for clarity.\nmodel\n=\nFastText\n(\nsentences\n=\ncorpus\n,\nvector_size\n=\n10\n,\nwindow\n=\n3\n,\nmin_count\n=\n1\n)\n# Common word\nprint\n(\n\"\nVector for\n'\ncat\n'\n:\n\"\n,\nmodel\n.\nwv\n[\n'\ncat\n'\n])\n# Rare word (seen once)\nprint\n(\n\"\nVector for\n'\njumped\n'\n:\n\"\n,\nmodel\n.\nwv\n[\n'\njumped\n'\n])\n# Plausible word never seen in training\nprint\n(\n\"\nVector for\n'\njumping\n'\n:\n\"\n,\nmodel\n.\nwv\n[\n'\njumping\n'\n])\n# Made-up or misspelled word\nprint\n(\n\"\nVector for\n'\ncattz\n'\n:\n\"\n,\nmodel\n.\nwv\n[\n'\ncattz\n'\n])\nEnter fullscreen mode\nExit fullscreen mode\nKey points:\n“jumping” and “cattz” were never in the training set. But FastText can produce vectors for them, by combining vector pieces from their n-grams.\nClassic Word2Vec or GloVe models cannot handle such words—they only know the words seen during training.\nWhen (and When Not) to Use FastText\nFastText’s subword approach is most useful when:\nYour text is messy, with typos, slang, or creative spelling (like in chat logs or social media).\nYou are working with languages that have many word forms due to rich grammar (morphology).\nYour domain or dataset has many rare, specialized words.\nSituations where FastText may be less helpful:\nFastText is slower. At prediction time, it needs to look up and combine several vectors for each word.\nSometimes, the similarities it finds are harder to explain, since words sharing n-grams might not actually be related in meaning.\nFor clean, well-edited text in a language like English, and when you have plenty of training data, simpler embeddings like Word2Vec or GloVe are often sufficient.\nIf your data is human-authored, noisy, or has lots of words outside your training set, FastText is a strong choice. For very large projects, it’s worth testing both classic and subword-based embeddings to see what works better for your needs.\nKey Takeaways\nWord embeddings translate words into numerical vectors capturing semantic similarity.\nClassic embeddings like one-hot cannot capture word relationships or handle unseen words.\nFastText creates embeddings using character n-grams, enabling vector creation for out-of-vocabulary words.\nSubword embeddings are robust to typos, rare words, and morphologically rich languages.\nFastText trades some speed and interpretability for improved handling of noisy, diverse data.\nTry It Yourself\nWrite a list of 5 words, including at least one made-up word and one misspelled word. Train a small FastText model on a simple dataset using Gensim, then extract vectors for your test words. Compare the vectors of the made-up and misspelled words to those of semantically or orthographically similar real words, and note how well FastText handles these out-of-vocabulary inputs.\nFurther Resources\n🎥\nFastText Explained | Subword Embeddings in NLP\n📘\nFastText embeddings, Word representations tutorial (fastText official documentation)\n📄\nEnriching Word Vectors with Subword Information\n📄\nIntro to FastText Subword Embeddings (Kaggle notebook)\nComing up on Day 15:\nVisualizing Embeddings with t-SNE and PCA",
  "metadata": {
    "title": "Day 14: FastText and Subword Embeddings - DEV Community",
    "description": "What Are Word Embeddings?   Words like \"cat,\" \"run,\" or \"happiness\" are just symbols to a... Tagged with ai, nlp, genai, 100daysofcode.",
    "keywords": "ai, nlp, genai, 100daysofcode, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/priyeshdave6/day-14-fasttext-and-subword-embeddings-34ed",
    "og:title": "Day 14: FastText and Subword Embeddings",
    "og:description": "What Are Word Embeddings?   Words like \"cat,\" \"run,\" or \"happiness\" are just symbols to a...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fipykjh23ms6miz0f0uws.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Day 14: FastText and Subword Embeddings",
    "twitter:description": "What Are Word Embeddings?   Words like \"cat,\" \"run,\" or \"happiness\" are just symbols to a...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fipykjh23ms6miz0f0uws.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/af17b80f6cb0/code/body-000.txt",
      "chars": 19,
      "language": "text"
    },
    {
      "file": "attachments/af17b80f6cb0/code/body-001.txt",
      "chars": 713,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=-K5QQIMV5F4",
      "status": "ok",
      "local_path": "attachments/af17b80f6cb0/videos/-K5QQIMV5F4.mp4",
      "bytes": 10935914,
      "valid": true,
      "height": 720,
      "width": 1280,
      "duration": "555.561000"
    }
  ],
  "links": [
    {
      "href": "https://www.youtube.com/watch?v=-K5QQIMV5F4",
      "label": "FastText Explained | Subword Embeddings in NLP",
      "type": "a"
    },
    {
      "href": "https://fasttext.cc/docs/en/unsupervised-tutorial.html",
      "label": "FastText embeddings, Word representations tutorial (fastText official documentation)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1607.04606",
      "label": "Enriching Word Vectors with Subword Information",
      "type": "a"
    },
    {
      "href": "https://www.kaggle.com/code/lorenzoscaturchio/intro-to-fasttext-subword-embeddings",
      "label": "Intro to FastText Subword Embeddings (Kaggle notebook)",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:10Z",
  "source_url": "https://dev.to/priyeshdave6/day-14-fasttext-and-subword-embeddings-34ed",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1607.04606",
      "bytes": 2374847,
      "reused": false,
      "local_path": "papers/repair/f1838fe5efe89dedd7be.pdf"
    }
  ],
  "actual_video_files": [
    {
      "local_path": "attachments/af17b80f6cb0/videos/-K5QQIMV5F4.mp4",
      "bytes": 10935914,
      "valid": true,
      "height": 720,
      "width": 1280,
      "duration": "555.561000",
      "status": "ok"
    }
  ],
  "body_chars": 7821,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/af17b80f6cb0/pages/source.html",
  "local_html": "attachments/af17b80f6cb0/pages/article.html",
  "body_base_path": "attachments/af17b80f6cb0/pages/",
  "body_markdown_path": "attachments/af17b80f6cb0/pages/article.md"
}