{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Hamel Husain / Parlance Labs",
  "title_listed": "Tokenization Gotchas",
  "keywords": "分词,词元,陷阱",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://hamel.dev/notes/llm/finetuning/tokenizer_gotchas.html",
  "publish_date": "2023-12",
  "corpus_stage": "数据处理",
  "notes": "分词器的常见陷阱",
  "id": "5c7de93841ff",
  "collected_at": "2026-10-02T14:29:04",
  "title": "Tokenization Gotchas – Hamel’s Blog",
  "body_markdown": "Tokenization Gotchas\nFootguns with tokenizers and inferencing LLMs\nBackground\nLots of people experience fiddly behavior when using LLMs. For example:\nUnironically I found this to be very helpful when prompting LLMs. Giving them spaces and new lines\npic.twitter.com/vVuxcCuDzB\n— anton (\n@abacaj\n)\nNovember 24, 2023\nIf you aren’t careful, these can be very hard to debug. This is because of the subtle ways tokenizers work that is not always easy to see by looking at the text.\nExample\nThe below example demonstrates how things can get confusing and can drift between training and inference time.\nfrom\ntransformers\nimport\nAutoTokenizer\nfrom\nfunctools\nimport\npartial\nmodel_id\n=\n'Open-Orca/Mistral-7B-OpenOrca'\ntok\n=\nAutoTokenizer.from_pretrained(model_id)\nenc\n=\npartial(tok.encode, add_special_tokens\n=\nFalse\n)\ndec\n=\npartial(tok.decode)\nMany frameworks do prompt construction by concatenating tokens\nPopular frameworks like\naxolotl\nconstruct prompts by concatenating tokens instead of strings.\n1\nIt is reasonable to decode the training data to check what the prompt template is:\nFor example, a prompt may be constructed like this:\naxolotl\n=\nenc(\n'Ok\n\\n\n'\n)\n+\nenc(\n'<|im_start|>'\n)\nprint\n(dec(axolotl))\nOk\n<|im_start|>\nLet’s say you have an inference server\nIt’s common for inference servers to assemble the prompt for you. The below looks like it should be fine, right?\ndef\ninf_server(inp):\nreturn\nf'\n{\ninp\n}\n\\n\n<|im_start|>'\nsrv\n=\ninf_server(\n'Ok'\n)\nprint\n(srv)\nOk\n<|im_start|>\nDrift between your server and the way the model is trained\nWrong! Notice the difference in the decoding of the prompt vs the training data. This is a subtle problem that can be hard to debug.\nprint\n(\nf'axolotl training data:\n{\naxolotl\n}\n'\n)\nprint\n(\nf\"your server's decoding:\n{\nenc(srv)\n}\n\"\n)\naxolotl training data:  [6504, 13, 32001]\nyour server's decoding: [6504, 32001]\nSolutions\n1. Decode your inference data\nDecode your inference data right before your forward pass. For example, you’ll notice the newline is missing if you do this. This is one way to tell that something fishy is going on.\ndec(enc(srv))\n'Ok<|im_start|>'\n2. Use HF chat templating\nUse the new HuggingFace\nchat template\nwhen possible. This will help avoid these issues (however, I would still check using method #1 to be sure!). Related GitHub Issue\ncomment\n.\nExample: Axolotl vs. HuggingFace Chat Templates\nThis is real example of how tokenization drift can bite you.\nChat Template From HuggingFace\nfrom\ntransformers\nimport\nAutoTokenizer\ntokenizer\n=\nAutoTokenizer.from_pretrained(\n\"NousResearch/Llama-2-7b-chat-hf\"\n)\nchat\n=\n[\n{\n\"role\"\n:\n\"system\"\n,\n\"content\"\n:\n\"lorem\"\n},\n{\n\"role\"\n:\n\"user\"\n,\n\"content\"\n:\n\"abc\"\n},\n{\n\"role\"\n:\n\"assistant\"\n,\n\"content\"\n:\n\"ipsum\"\n},\n{\n\"role\"\n:\n\"user\"\n,\n\"content\"\n:\n\"123\"\n},\n{\n\"role\"\n:\n\"assistant\"\n,\n\"content\"\n:\n\"sit\"\n},\n]\nids\n=\ntokenizer.apply_chat_template(chat)\nprint\n(tokenizer.decode(ids))\n<s>[INST] <<SYS>>\nlorem\n<</SYS>>\n\nabc [/INST] ipsum</s><s>[INST] 123 [/INST] sit</s>\nSame thing decoded from Axolotl (with a space after\n<s>\n)\nGot the token ids from\nthis test\n.\naxolotl_ids\n=\n[\n1\n,\n518\n,\n25580\n,\n29962\n,\n3532\n,\n14816\n,\n29903\n,\n6778\n,\n13\n,\n29880\n,\n3668\n,\n13\n,\n29966\n,\n829\n,\n14816\n,\n29903\n,\n6778\n,\n13\n,\n13\n,\n10736\n,\n518\n,\n29914\n,\n25580\n,\n29962\n,\n23421\n,\n2\n,\n1\n,\n518\n,\n25580\n,\n29962\n,\n29871\n,\n29896\n,\n29906\n,\n29941\n,\n518\n,\n29914\n,\n25580\n,\n29962\n,\n7845\n,\n2\n]\nprint\n(tokenizer.decode(axolotl_ids))\n<s> [INST] <<SYS>>\nlorem\n<</SYS>>\n\nabc [/INST] ipsum</s><s> [INST] 123 [/INST] sit</s>\nLet’s decode HF tokens one at a time\nfor\ni\nin\nids[:\n9\n]:\nprint\n(\nf'\n{\ni\n}\n:\n{\ntokenizer\n.\ndecode(i)\n}\n'\n)\n1: <s>\n29961: [\n25580: INST\n29962: ]\n3532: <<\n14816: SY\n29903: S\n6778: >>\n13:\nLet’s decode Axolotl tokens one at a time\nSee the second token\n518\nthis is a mismatch with the HF Chat template which is\n29961\nfor\ni\nin\naxolotl_ids[:\n9\n]:\nprint\n(\nf'\n{\ni\n}\n:\n{\ntokenizer\n.\ndecode(i)\n}\n'\n)\n1: <s>\n518: [\n25580: INST\n29962: ]\n3532: <<\n14816: SY\n29903: S\n6778: >>\n13:\nWhy does this happen?\nAxolotl assembles prompts in token space rather than string space.\ntokenizer.encode(\n'<s>'\n, add_special_tokens\n=\nFalse\n)\n+\ntokenizer.encode(\n'[INST]'\n, add_special_tokens\n=\nFalse\n)\n[1, 518, 25580, 29962]\nHF Chat templates interpolate strings instead\ntokenizer.encode(\n'<s>[INST]'\n, add_special_tokens\n=\nFalse\n)\n[1, 29961, 25580, 29962]\nOther Examples\nThese are other examples of people being bitten by drift between differences in tokenization between training and inference time:\nThis\nGitHub Issue\n.\nThis\nTweet\n.\nFootnotes\nThis is for good reason, as masking must also be done at the token level.\n↩︎",
  "metadata": {
    "title": "Tokenization Gotchas – Hamel’s Blog",
    "description": "Footguns with tokenizers and inferencing LLMs",
    "keywords": null,
    "author": null,
    "og:type": "article",
    "og:title": "Tokenization Gotchas – Hamel’s Blog",
    "og:description": "Footguns with tokenizers and inferencing LLMs",
    "og:image": "https://user-images.githubusercontent.com/1483922/208359430-f55d7503-3a98-4875-a35c-16314c9439d0.png",
    "og:site_name": "Hamel’s Blog",
    "og:locale": "en_US",
    "twitter:title": "Tokenization Gotchas – Hamel’s Blog",
    "twitter:description": "Footguns with tokenizers and inferencing LLMs",
    "twitter:image": "https://user-images.githubusercontent.com/1483922/208359430-f55d7503-3a98-4875-a35c-16314c9439d0.png",
    "twitter:creator": "@HamelHusain",
    "twitter:site": "@HamelHusain",
    "twitter:card": "summary_large_image"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/5c7de93841ff/code/body-000.txt",
      "chars": 157,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-001.txt",
      "chars": 77,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-002.txt",
      "chars": 63,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-003.txt",
      "chars": 15,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-004.txt",
      "chars": 90,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-005.txt",
      "chars": 15,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-006.txt",
      "chars": 88,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-007.txt",
      "chars": 79,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-008.txt",
      "chars": 13,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-009.txt",
      "chars": 16,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-010.txt",
      "chars": 410,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-011.txt",
      "chars": 84,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-012.txt",
      "chars": 356,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-013.txt",
      "chars": 86,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-014.txt",
      "chars": 58,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-015.txt",
      "chars": 79,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-016.txt",
      "chars": 66,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-017.txt",
      "chars": 77,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-018.txt",
      "chars": 104,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-019.txt",
      "chars": 22,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-020.txt",
      "chars": 55,
      "language": "text"
    },
    {
      "file": "attachments/5c7de93841ff/code/body-021.txt",
      "chars": 24,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.googletagmanager.com/ns.html?id=GTM-PKGWQMKL",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://t.co/vVuxcCuDzB",
      "label": "pic.twitter.com/vVuxcCuDzB",
      "type": "a"
    },
    {
      "href": "https://twitter.com/abacaj/status/1728190808191537604?ref_src=twsrc%5Etfw",
      "label": "November 24, 2023",
      "type": "a"
    },
    {
      "href": "https://github.com/OpenAccess-AI-Collective/axolotl",
      "label": "axolotl",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/transformers/chat_templating",
      "label": "chat template",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers/issues/25304",
      "label": "comment",
      "type": "a"
    },
    {
      "href": "https://github.com/OpenAccess-AI-Collective/axolotl/pull/952/files",
      "label": "this test",
      "type": "a"
    },
    {
      "href": "https://twitter.com/johnowhitaker/status/1732097798286475578",
      "label": "Tweet",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:11:31",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:44Z",
  "source_url": "https://hamel.dev/notes/llm/finetuning/tokenizer_gotchas.html",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 4551,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/5c7de93841ff/pages/source.html",
  "local_html": "attachments/5c7de93841ff/pages/article.html",
  "body_base_path": "attachments/5c7de93841ff/pages/",
  "body_markdown_path": "attachments/5c7de93841ff/pages/article.md"
}