{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Bala Priya C",
  "title_listed": "N-Gram Language Models - A Beginner's Guide",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/balapriya/understanding-n-gram-language-models-3g72",
  "publish_date": "2021-02-28",
  "corpus_stage": "",
  "notes": "",
  "id": "f23fe5ef1428",
  "collected_at": "2026-10-02T14:01:52",
  "title": "N-Gram Language Models - A Beginner's Guide - DEV Community",
  "body_markdown": "Have you ever guessed what the next sentence in the paragraph you’re reading would likely talk about? Have you ever noticed that while reading, you almost always know the next word in the sentence?\nWell, the answer to these questions is definitely\nYes\n!\nAs humans, we’re bestowed with the ability to read, understand languages and\ninterpret contexts\n, and can almost always predict the next word in a text, based on what we’ve read so far.\nCan we make a machine learning model do the same?\nOh yeah! We very well can! And we already use such models everyday, such as the\nautocomplete\nfeature in search engines and messaging apps.\nIn the context of Natural Language Processing, the task of predicting which word comes next is called\nLanguage Modeling\n.\nLet’s take a simple example,\nThe students opened their _______.\nWhat are the possible words that we can fill the blank with?\nBooks📗 📒📚\nNotes📖\nLaptops👩🏽‍💻\nMinds💡🙂\nExams📑❔\nWell, the list goes on. Wait…why did we think of these words as the best choices, rather than ‘opened their\ndoors\nor\nwindows’\n?\nIt’s because we had the word students, and given the context\n‘students’\n, the words such as books, notes and laptops seem more likely and therefore have a higher probability of occurrence than the words doors and windows.\nTypically, this\nprobability\nis what a language model aims at computing. Over the next few minutes, we’ll see the notion of n-grams, a very effective and popular traditional NLP technique, widely used before deep learning models became popular.\nWhat Does a Language Model Do?\nDescribing in formal terms,\nGiven a text corpus with vocabulary\nV\n,\nGiven a sequence of words,\nx(1),x(2),…,x(t)\n,\nA language model essentially computes the\nprobability distribution of the next word\nx(t+1)\n.\nA language model, thus, assigns a probability to a piece of text. The probability can be expressed using chain rule as the product of the following probabilities.\nProbability of the first word being\nx(1)\nProbability of the second word being\nx(2)\ngiven that the first word is\nx(1)\nProbability of the third word being\nx(3)\ngiven that the first two words are\nx(1)\nand\nx(2)\nIn general, the conditional probability that\nx(i)\nis word i, given that the first (i-1) words are\nx(1),x(2),…,x(i-1)\n.\nThe probability of the text according to the language model is:\nTo Learn a Language Model, Learn N-Grams!\nAn n-gram is a\nchunk of n consecutive words\n.\nFor our example, The students opened their _______, the following are the n-grams for n=1,2,3 and 4.\nuni\ngrams: “the”, “students”, “opened”, ”their”\nbi\ngrams: “the students”, “students opened”, “opened their”\ntri\ngrams: “the students opened”, “students opened their”\n4\n- grams: “the students opened their”\nIn an n-gram language model, we make an assumption that the word\nx(t+1)\ndepends only on the previous (n-1) words. The idea is to collect how frequently the n-grams occur in our corpus and use it to predict the next word.\nThis equation, on applying the definition of conditional probability yields,\nHow Do We Compute These Probabilities?\nTo compute the probabilities of these n-grams and n-1 grams, we just go ahead and start counting them in a large text corpus!\nThe Probability of n-gram/Probability of (n-1) gram is given by:\nLet’s learn a 4-gram language model for the example,\nAs the proctor started the clock, the students opened their _____\nIn learning a 4-gram language model, the next word (the word that fills up the blank) depends only on the previous 3 words. If\nw\nis the word that goes into the blank, then we compute the conditional probability of the word\nw\nas follows:\nIn the above example, let us say we have the following:\n\"\nstudents opened their\n\"\noccurred\n1000\ntimes\n\"\nstudents opened their books\n\"\noccurred\n400\ntimes\n->\nP\n(\nbooks\n/\nstudents\nopened\ntheir\n)\n=\n0.4\n\"\nstudents opened their exams\n\"\noccurred\n200\ntimes\n->\nP\n(\nexams\n/\nstudents\nopened\ntheir\n)\n=\n0.2\nEnter fullscreen mode\nExit fullscreen mode\nThe language model would predict the word\nbooks\n; But given the context, is\nbooks\nreally the right choice? Wouldn’t the word\nexams\nbe a better fit? Recall that we have,\nAs the proctor started the clock, the students opened their _____\nShould we really have discarded the context ‘\nproctor\n’?\nLooks like we shouldn’t have. This leads us to understand some of the problems associated with n-grams.\nDisadvantages of N-gram Language Models\nProblem of Sparsity\nWhat if “students opened their” never occurred in the corpus?\nThe count term in the denominator would go to zero!\nIf the (n-1) gram never occurred in the corpus, then we cannot compute the probabilities. In that case, we may have to revert to using “opened their” instead of “students opened their”, and this strategy is called\nback-off\n.\nWhat if “students opened their w” never occurred in the corpus?\nThe count term in the numerator would go to zero!\nIf word\nw\nnever appeared after the n-1 gram, then we may have to add a small factor\ndelta\nto the count, that accounts for all words in the vocabulary\nV\n.This is called\n‘smoothing’\n. Sparsity problem increases with increasing\nn\n. In practice,\nn\ncannot be greater than 5.\nProblem of Storage\nAs we need to store count for all possible n-grams in the corpus, increasing n or increasing the size of the corpus, both tend to become storage-inefficient.\nHowever, n-gram language models can also be used for text generation; a tutorial on generating text using such n-grams can be found in reference[2] given below.\nReferences\n[1]\nCS224n: Natural Language Processing with Deep Learning\n[2]\nNLP for Hackers\n[3] All equations are from the\nlecture slides of CS224n\nCover Image:\nPhoto by\nBrett Jordan\non\nUnsplash",
  "metadata": {
    "title": "N-Gram Language Models - A Beginner's Guide - DEV Community",
    "description": "Have you ever guessed what the next sentence in the paragraph you’re reading would likely talk... Tagged with machinelearning, nlp, python, beginners.",
    "keywords": "machinelearning, nlp, python, beginners, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/balapriya/understanding-n-gram-language-models-3g72",
    "og:title": "N-Gram Language Models - A Beginner's Guide",
    "og:description": "Have you ever guessed what the next sentence in the paragraph you’re reading would likely talk...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F6jwcjuwqmtnt96mbrna8.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "N-Gram Language Models - A Beginner's Guide",
    "twitter:description": "Have you ever guessed what the next sentence in the paragraph you’re reading would likely talk...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F6jwcjuwqmtnt96mbrna8.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/f23fe5ef1428/code/body-000.txt",
      "chars": 222,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fv0izfsfunm1qgzdb9394.jpeg",
      "bytes": 30118,
      "reused": true,
      "local_path": "attachments/f23fe5ef1428/images/002.jpeg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fv0izfsfunm1qgzdb9394.jpeg",
      "alt": "Probability distribution of the next word x(t+1) given x(1)…x(t)",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpukyi64wx92ndnw34gis.jpeg",
      "bytes": 25592,
      "reused": true,
      "local_path": "attachments/f23fe5ef1428/images/003.jpeg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpukyi64wx92ndnw34gis.jpeg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fh0yx9v1zfdohketcswss.jpg",
      "bytes": 9924,
      "reused": true,
      "local_path": "attachments/f23fe5ef1428/images/004.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fh0yx9v1zfdohketcswss.jpg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ftozjnkv2rgesat94eftt.jpeg",
      "bytes": 38724,
      "reused": true,
      "local_path": "attachments/f23fe5ef1428/images/005.jpeg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ftozjnkv2rgesat94eftt.jpeg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkdh8zmbpmei96xpqg7ra.jpeg",
      "bytes": 12823,
      "reused": true,
      "local_path": "attachments/f23fe5ef1428/images/006.jpeg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkdh8zmbpmei96xpqg7ra.jpeg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fiqnm0obhwerc2gl2k1jh.jpg",
      "bytes": 13441,
      "reused": true,
      "local_path": "attachments/f23fe5ef1428/images/007.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fiqnm0obhwerc2gl2k1jh.jpg",
      "alt": "Alt Text",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fj59b03uks0rs6nbeo4n8.jpeg",
      "bytes": 31200,
      "reused": true,
      "local_path": "attachments/f23fe5ef1428/images/008.jpeg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fj59b03uks0rs6nbeo4n8.jpeg",
      "alt": "Alt Text",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fv0izfsfunm1qgzdb9394.jpeg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpukyi64wx92ndnw34gis.jpeg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fh0yx9v1zfdohketcswss.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ftozjnkv2rgesat94eftt.jpeg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkdh8zmbpmei96xpqg7ra.jpeg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fiqnm0obhwerc2gl2k1jh.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fj59b03uks0rs6nbeo4n8.jpeg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/",
      "label": "CS224n: Natural Language Processing with Deep Learning",
      "type": "a"
    },
    {
      "href": "https://nlpforhackers.io/language-models/",
      "label": "NLP for Hackers",
      "type": "a"
    },
    {
      "href": "https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/slides/cs224n-2019-lecture06-rnnlm.pdf",
      "label": "lecture slides of CS224n",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/@brett_jordan?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Brett Jordan",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/s/photos/words?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Unsplash",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:14Z",
  "source_url": "https://dev.to/balapriya/understanding-n-gram-language-models-3g72",
  "papers": [
    {
      "status": "ok",
      "url": "https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/slides/cs224n-2019-lecture06-rnnlm.pdf",
      "bytes": 2085608,
      "reused": false,
      "local_path": "papers/repair/a6b3fae066e2a164c010.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 5630,
  "expected_body_images": 7,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/f23fe5ef1428/pages/source.html",
  "local_html": "attachments/f23fe5ef1428/pages/article.html",
  "body_base_path": "attachments/f23fe5ef1428/pages/",
  "body_markdown_path": "attachments/f23fe5ef1428/pages/article.md"
}