{
  "source_file": "0930",
  "source_platform": "Beehiiv",
  "author": "Akshay Pachaar / ML Spring",
  "title_listed": "Understanding Tokenization in NLP",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://mlspring.beehiiv.com/p/understanding-tokenization-innlp",
  "publish_date": "2023-07",
  "corpus_stage": "",
  "notes": "",
  "id": "04dbe7fac76e",
  "collected_at": "2026-10-02T14:12:19",
  "title": "Tokenization in NLP & LLMs",
  "body_markdown": "0\n0\nML Spring\nPosts\nUnderstanding Tokenization in NLP.\nUnderstanding Tokenization in NLP.\nA hands on tutorial with code! 🚀\nAkshay Pachaar\nJuly 05, 2023\nUnderstanding Tokenization in NLP!\nTokenization is the process of breaking a stream of text into words, phrases, symbols, or other meaningful elements called tokens. Now these tokens can be numerically encoded & given as an input to model.\nThe performance of your model largely depends on how well you are able to tokenize the inputs and this makes tokenization an essential preprocessing step when working on NLP tasks.\nTo better understand tokenization, let’s talk about different way a raw piece of text can be converted into tokens and weigh the advantages and disadvantages of each method.\nCharacter Tokenization\nWord Tokenization\nSubword Tokenization\nEntire Dataset Tokenization\nThis would be a hand-on exercise so get you Jupyter Notebooks running!!\n📒\n🚀\nCharacter Tokenization:\nThe simplest tokenization strategy would be to break down the text into individual character tokens and feed them to the model. And implementing this in python doesn’t get easier.\nCheck this out 👇\nraw_text = \"We love NLP!\"\ntokens = list(raw_text)\nprint(tokens)\n\n👉 ['W', 'e', ' ', 'l', 'o', 'v', 'e', ' ', 'N', 'L', 'P', '!']\nNow, our model needs each token to be converted into an integer.\nHere’s how we do it 👇\n# Numerical encoding of individual character\n\ntoken2idx = {char: idx for idx, char in enumerate(sorted(set(tokens)))}\nprint(token2idx)\n\n👉 {' ': 0, '!': 1, 'L': 2, 'N': 3, 'P': 4, 'W': 5, 'e': 6, 'l': 7, 'o': 8, 'v': 9}\n\n# Using token2idx to map our tokenized text to integers\n\ninteger_tokens = [token2idx[token] for token in tokens]\nprint(integer_tokens)\n\n👉 [5, 6, 0, 7, 8, 9, 6, 0, 3, 2, 4, 1]\nWe now have each of our token mapped to a unique integer. The next step would be to one-hot encode each of these integers.\nCheck this out 👇\nSubscribe to keep reading\nThis content is free, but you must be subscribed to ML Spring to continue reading.\nAlready a subscriber?\nSign in\n.\nNot now\nReply\nNewest first\nAdd your comment\nLogin\nLogin\nor\nSubscribe\nto participate\n.",
  "metadata": {
    "title": "Tokenization in NLP & LLMs",
    "description": "A hands on tutorial with code! 🚀",
    "keywords": "data science, programming, machine learning",
    "author": "Akshay Pachaar",
    "og:type": "article",
    "og:url": "https://mlspring.beehiiv.com/p/understanding-tokenization-innlp",
    "og:title": "Tokenization in NLP & LLMs",
    "og:description": "A hands on tutorial with code! 🚀",
    "og:site_name": "ML Spring",
    "og:image": "https://beehiiv-images-production.s3.amazonaws.com/uploads/publication/thumbnail/39426994-01e5-416d-ad8a-4427d2705032/landscape_mlspring_thumbnail.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "A hands on tutorial with code! 🚀",
    "twitter:url": "https://mlspring.beehiiv.com/p/understanding-tokenization-innlp",
    "twitter:title": "Tokenization in NLP & LLMs",
    "twitter:description": "A hands on tutorial with code! 🚀",
    "twitter:image": "https://beehiiv-images-production.s3.amazonaws.com/uploads/publication/thumbnail/39426994-01e5-416d-ad8a-4427d2705032/landscape_mlspring_thumbnail.png",
    "twitter:card": "summary_large_image",
    "article:author": "Akshay Pachaar"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/04dbe7fac76e/code/body-000.txt",
      "chars": 127,
      "language": "text"
    },
    {
      "file": "attachments/04dbe7fac76e/code/body-001.txt",
      "chars": 394,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/user/profile_picture/365b6afb-67c5-48c4-ade3-f57b97ccf42a/thumb_akshay.jpeg",
      "bytes": 7364,
      "reused": true,
      "local_path": "attachments/04dbe7fac76e/images/000.jpeg",
      "src": "https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/user/profile_picture/365b6afb-67c5-48c4-ade3-f57b97ccf42a/thumb_akshay.jpeg",
      "alt": "Author",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/3962a508-d823-4f32-8c56-86c30e07f24d/we_love_nlp__1_.png",
      "bytes": 25466,
      "reused": true,
      "local_path": "attachments/04dbe7fac76e/images/001.png",
      "src": "https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/3962a508-d823-4f32-8c56-86c30e07f24d/we_love_nlp__1_.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://mlspring.beehiiv.com/",
      "label": "ML Spring",
      "type": "a"
    },
    {
      "href": "https://mlspring.beehiiv.com/authors/akshay-pachaar",
      "label": "Akshay Pachaar",
      "type": "a"
    },
    {
      "href": "https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmlspring.beehiiv.com%2Fp%2Funderstanding-tokenization-innlp",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://twitter.com/intent/tweet?text=A+hands+on+tutorial+with+code%21+%F0%9F%9A%80+&url=https%3A%2F%2Fmlspring.beehiiv.com%2Fp%2Funderstanding-tokenization-innlp&via=akshay_pachaar",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.threads.net/intent/post?text=A+hands+on+tutorial+with+code%21+%F0%9F%9A%80++https%3A%2F%2Fmlspring.beehiiv.com%2Fp%2Funderstanding-tokenization-innlp",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/sharing/share-offsite?url=https%3A%2F%2Fmlspring.beehiiv.com%2Fp%2Funderstanding-tokenization-innlp",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://mlspring.beehiiv.com/login?postSlug=understanding-tokenization-innlp",
      "label": "Sign in .",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:09Z",
  "source_url": "https://mlspring.beehiiv.com/p/understanding-tokenization-innlp",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 2110,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/04dbe7fac76e/pages/source.html",
  "local_html": "attachments/04dbe7fac76e/pages/article.html",
  "body_base_path": "attachments/04dbe7fac76e/pages/",
  "body_markdown_path": "attachments/04dbe7fac76e/pages/article.md"
}