{ "source_file": "0930", "source_platform": "Dev.to", "author": "Javier Aguirre", "title_listed": "Tokenization in NLP: How Machines Break Down Text", "keywords": "", "domain": "文本处理", "content_type": "技术教程", "is_paid_flag": "否", "url": "https://dev.to/javiagu13/tokenization-in-nlp-how-machines-break-down-text-5hej", "publish_date": "2026-09-04", "corpus_stage": "", "notes": "", "id": "f1340b2ba510", "collected_at": "2026-10-02T13:54:16", "title": "Tokenization in NLP: How Machines Break Down Text - DEV Community", "body_markdown": "Before a computer can understand a single word you write, it has to do something much more basic.\nIt has to figure out where one piece of text ends and the next begins.\nThat process is called\ntokenization\n, and it’s the first step in almost every NLP system ever built. Whether you’re using a spam filter, a search engine, or ChatGPT, tokenization happened before anything else. It’s the foundation everything else is built on.\nThis post explains what tokenization is, why it matters, and how it has evolved from simple word splitting into the sophisticated subword methods that power modern AI. Along the way, we’ll also cover the related concepts you’ll encounter whenever you work with text: stemming, lemmatization, and n-grams.\nWhat Is Tokenization?\nTokenization\nis the process of breaking text into smaller units called tokens.\nA token is just a piece of text. It might be a word, a punctuation mark, a part of a word, or even a single character. The important thing is that tokens are the units a model actually works with. Before any analysis, translation, or generation can happen, the raw text needs to be split into these pieces.\nTake this sentence:\n“NLP is surprisingly powerful.”\nAfter word tokenization, it becomes:\n[“NLP”, “is”, “surprisingly”, “powerful”, “.”]\nThat’s it. Five tokens. The model now has something it can process.\nSimple in concept. But the details turn out to matter enormously.\nWord tokenization splits a sentence into individual tokens including punctuation\nWhy Tokenization Is Harder Than It Looks\nYou might be thinking: can’t you just split on spaces?\nFor English, that gets you most of the way there. But language is messier than that.\nWhat do you do with\n“don’t”\n? Is it one token or two (\n“do”\nand\n“n’t”\n)? What about\n“New York”\n? Splitting on spaces gives you two tokens, but it’s one concept. What about emojis, URLs, email addresses, or code snippets mixed into text?\nAnd that’s just English. Languages like Chinese and Japanese have no spaces between words at all. Arabic and Hebrew write without vowels. German creates long compound words that are really several concepts joined together.\nTokenization that works well across all of these is a genuinely hard problem. And how you solve it has downstream consequences for everything the model learns.\nWord Tokenization\nThe most intuitive approach is\nword tokenization\n: split the text into individual words.\nThis works well for clean, formal text. It’s fast, easy to understand, and produces tokens that map directly to human intuitions about language.\nThe limitation is vocabulary size. If you train a model on word tokens, it can only handle words it has seen before. Give it\n“unimaginably”\nwhen it’s only ever seen\n“imagine”\nand\n“imaginable”\n, and it’s stuck. This is called the\nout-of-vocabulary (OOV) problem\n, and it’s one of the main reasons word tokenization has been largely superseded for modern AI systems.\nStemming and Lemmatization: Reducing Words to Their Core\nBefore diving into more advanced tokenization, it’s worth understanding two related text processing techniques you’ll encounter constantly: stemming and lemmatization.\nBoth answer the same question: when should\n“running”\n,\n“runs”\n, and\n“ran”\nbe treated as the same word?\nStemming\nStemming\nis the blunter approach. It chops off the ends of words using simple rules until you’re left with a root form called a stem.\n“Running”\nbecomes\n“run”\n.\n“Fishing”\nbecomes\n“fish”\n.\n“Better”\nbecomes\n“better”\n(stemming can’t handle irregular forms).\nIt’s fast and simple, but it’s crude. The stem doesn’t have to be a real word.\n“Arguing”\nmight become\n“argu”\n.\n“Studies”\nmight become\n“studi”\n. That’s fine for some applications, like topic modelling or search indexing, where you just need to group related words together. It falls apart when you need the actual meaning of a word.\nLemmatization\nLemmatization\nis the smarter approach. Instead of blindly chopping characters, it uses vocabulary and grammar rules to find the true base form of a word, called the lemma.\nLemmatization understands that\n“better”\nis a form of\n“good”\n, something stemming can never figure out. It’s slower and requires more linguistic knowledge, but the output is always a real, meaningful word.\nThe Key Difference:\n“Stemming is fast and crude. Lemmatization is slower and correct. Choose based on what your task actually needs.”\nWhen to use each:\nStemming:\nsearch indexing, topic modelling, situations where speed matters and rough grouping is enough\nLemmatization:\nsentiment analysis, question answering, any task where word meaning matters\nSubword Tokenization: The Modern Solution\nNeither word tokenization nor character tokenization is quite right for modern AI. Word tokenization has the OOV problem. Character tokenization produces sequences so long they’re computationally impractical.\nThe solution the field landed on is\nsubword tokenization\n: split words into pieces, but not all the way down to individual characters.\nThe idea is elegant. Common words stay as single tokens. Rare or unknown words get split into recognisable subword pieces.\n“Tokenization”\nmight become\n“token”\n+\n“ization”\n.\n“Unbelievable”\nmight become\n“un”\n+\n“believ”\n+\n“able”\n.\nThis gives the model the best of both worlds. It can handle any word, including ones it has never seen, because it can always fall back to subword pieces. And it keeps vocabulary size manageable.\nSubword tokenization splits rare words into recognisable pieces while keeping common words intact\nByte Pair Encoding (BPE)\nByte Pair Encoding (BPE)\nis the most widely used subword tokenization algorithm. It powers GPT, GPT-2, GPT-3, GPT-4, and many other major models.\nThe algorithm works like this:\nStart with every character as its own token. Then repeatedly find the most common pair of adjacent tokens and merge them into a single new token. Repeat until you reach your desired vocabulary size.\nSo if\n“lo”\nand\n“w”\nappear together constantly, they get merged into\n“low”\n. If\n“low”\nand\n“er”\nappear together constantly, they get merged into\n“lower”\n. The algorithm keeps building up frequent sequences into single tokens.\nThe result is a vocabulary that naturally captures common words and common word fragments, while still being able to handle anything new by falling back to shorter pieces.\nWordPiece\nWordPiece\nis a similar approach used by BERT and many Google models. The main difference is in how it decides what to merge. BPE merges the most frequent pairs. WordPiece merges the pairs that increase the likelihood of the training data the most.\nIn practice, the results look similar. Subword pieces in WordPiece are often prefixed with\n##\nto indicate they’re continuations of a word.\n“playing”\nmight become\n“play”\n+\n“##ing”\n.\nSentencePiece\nSentencePiece\ntakes a slightly different approach: it treats the input as a raw stream of characters (including spaces) rather than pre-tokenized words. This makes it language-agnostic. It works equally well on English, Japanese, Chinese, or any other language without needing language-specific rules. It’s used in models like T5 and many multilingual systems.\nN-grams: Capturing Context Between Tokens\nOnce you have tokens, a natural next question is: does the order matter?\nAlmost always, yes.\n“Not good”\nmeans something very different from\n“good”\n.\n“Bank robbery”\nmeans something different from\n“robbery bank”\n.\nN-grams\nare a simple way to capture this. An n-gram is a sequence of n consecutive tokens.\nBy looking at n-grams instead of individual tokens, models can capture short-range context.\n“New York”\nas a bigram carries different meaning than\n“New”\nand\n“York”\nseparately.\nN-grams were the backbone of language modelling before neural networks. They’re still used in text classification, spam detection, and search systems where simplicity and speed matter more than capturing long-range context.\nThe limitation is obvious: n-grams only capture context within a short window. A trigram can’t know that the word at the start of a paragraph shapes the meaning of a word at the end. That’s what neural networks, and ultimately the Transformer, were built to solve.\nN-grams capture sequences of consecutive tokens: unigrams, bigrams, and trigrams\nHow Modern LLMs Tokenize Text\nIf you’ve ever used ChatGPT or Claude and noticed that pricing is measured in “tokens,” this is where that comes from.\nModern large language models use BPE or similar subword methods. Every piece of text you send gets broken into tokens before the model ever sees it. The model generates its response one token at a time.\nOn average, one token is roughly four characters in English, or about three-quarters of a word.\n“Tokenization”\nis two tokens.\n“Hello”\nis one. A typical paragraph is around 100 tokens.\nThis is also why LLMs sometimes handle unusual words or names strangely. If a word gets split into many subword pieces, the model has to work harder to reason about it as a coherent unit. The tokenization shapes what the model finds easy or difficult.\nPutting It Together\nHere’s a quick summary of the landscape:\nWord tokenization:\nsimple, intuitive, but can’t handle unknown words\nStemming:\nfast root-finding, crude, sometimes produces non-words\nLemmatization:\naccurate root-finding, slower, always produces real words\nSubword tokenization (BPE, WordPiece, SentencePiece):\nthe modern standard, handles any word, used in all major LLMs\nN-grams:\nsequences of tokens that capture short-range context, still used in classical NLP\nEach approach has its place. For production NLP systems today, subword tokenization is almost always the right choice. For understanding older systems or building lightweight classifiers, word tokens and n-grams still have their role.\nWhat Comes Next\nTokenization turns raw text into something a model can process. But once you have tokens, the next question is: what role does each one play in the sentence?\nThat’s what\nPart of Speech Tagging\ncovers. It’s the process of labelling each token with its grammatical role: noun, verb, adjective, and so on. It sounds simple, but it’s one of the foundational steps that allows NLP systems to understand the structure of language, not just the words themselves.\nIn the next post in this series, we’ll break down how POS tagging works, why it matters, and where it shows up in real NLP pipelines.\nLearn This at Fondra Labs\nThis post is part of our\nNLP Foundations series\n, where we cover everything from the basics of text processing to the architectures powering modern AI systems.\nAt\nFondra Labs\n, we teach AI from production reality, not hype. Every concept in this series is chosen because it matters in practice, not just in theory. Tokenization, embeddings, retrieval, evaluation: these are the building blocks that engineers actually use when they build real systems.\nIf you found this useful, explore the rest of the blog. We cover machine learning, deep learning, NLP, and the practical skills that bridge the gap between understanding AI and building with it.\nOriginally published at\nhttps://fondralabs.com\n.", "metadata": { "title": "Tokenization in NLP: How Machines Break Down Text - DEV Community", "description": "Before a computer can understand a single word you write, it has to do something much more... Tagged with ai, machinelearning, nlp.", "keywords": "ai, machinelearning, nlp, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/javiagu13/tokenization-in-nlp-how-machines-break-down-text-5hej", "og:title": "Tokenization in NLP: How Machines Break Down Text", "og:description": "Before a computer can understand a single word you write, it has to do something much more...", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvbpiti9yfawdvzd1by13.png", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "Tokenization in NLP: How Machines Break Down Text", "twitter:description": "Before a computer can understand a single word you write, it has to do something much more...", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvbpiti9yfawdvzd1by13.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkz3an4nvgl0hxtvul5fm.png", "bytes": 106438, "reused": true, "local_path": "attachments/f1340b2ba510/images/001.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkz3an4nvgl0hxtvul5fm.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcnxa5cegszjmoet3jrfk.png", "bytes": 28886, "reused": true, "local_path": "attachments/f1340b2ba510/images/002.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcnxa5cegszjmoet3jrfk.png", "alt": "Word tokenization splits a sentence into individual tokens including punctuation", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2dv1rv0h0on5kisawqtq.png", "bytes": 95622, "reused": true, "local_path": "attachments/f1340b2ba510/images/003.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2dv1rv0h0on5kisawqtq.png", "alt": "Subword tokenization splits rare words into recognisable pieces while keeping common words intact", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg5yfcnidas5ohmuh601y.png", "bytes": 97316, "reused": true, "local_path": "attachments/f1340b2ba510/images/004.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg5yfcnidas5ohmuh601y.png", "alt": "N-grams capture sequences of consecutive tokens: unigrams, bigrams, and trigrams", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkz3an4nvgl0hxtvul5fm.png", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcnxa5cegszjmoet3jrfk.png", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2dv1rv0h0on5kisawqtq.png", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg5yfcnidas5ohmuh601y.png", "label": "", "type": "a" }, { "href": "https://fondralabs.com/nlp-foundations.html", "label": "NLP Foundations series", "type": "a" }, { "href": "http://fondralabs.com/", "label": "Fondra Labs", "type": "a" }, { "href": "https://fondralabs.com/blog/nlp-foundations/tokenization-in-nlp-how-machines-break-down-text.html", "label": "https://fondralabs.com", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:48:56Z", "source_url": "https://dev.to/javiagu13/tokenization-in-nlp-how-machines-break-down-text-5hej", "papers": [], "actual_video_files": [], "body_chars": 11005, "expected_body_images": 4, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/f1340b2ba510/pages/source.html", "local_html": "attachments/f1340b2ba510/pages/article.html", "body_base_path": "attachments/f1340b2ba510/pages/", "body_markdown_path": "attachments/f1340b2ba510/pages/article.md" }