{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "Hugging Face Blog",
  "title_listed": "Tokenizers: the fastest tokenization library",
  "keywords": "分词器,词元,库",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://huggingface.co/blog/tokenizers",
  "publish_date": "2023-12",
  "corpus_stage": "数据处理",
  "notes": "Tokenizers分词库介绍",
  "id": "c0bb8359279a",
  "collected_at": "2026-10-02T15:23:57",
  "title": "Tokenization in Transformers v5: Simpler, Clearer, and More Modular",
  "body_markdown": "Back to Articles\nTokenization in Transformers v5: Simpler, Clearer, and More Modular\nPublished\n\t\t\t\t\tDecember 18, 2025\nUpdate on GitHub\nUpvote\n128\n+122\nIta Zaporozhets\nitazap\nFollow\nAritra Roy Gosthipaty\nariG23498\nFollow\nArthur Zucker\nArthurZ\nFollow\nSergio Paniego\nsergiopaniego\nFollow\nmerve\nmerve\nFollow\nPedro Cuenca\npcuenq\nFollow\nTransformers v5\nredesigns how tokenizers work. The\nbig tokenizers reformat\nseparates tokenizer design from trained vocabulary (much like how PyTorch separates neural network architecture from learned weights). The result is tokenizers you can\ninspect\n,\ncustomize\n, and\ntrain\nfrom scratch with far less friction.\nTL;DR: This blog explains how tokenization works in Transformers and why v5 is a major redesign, with clearer internals, a clean class hierarchy, and a single fast backend. It’s a practical guide for anyone who wants to understand, customize, or train model-specific tokenizers instead of treating them as black boxes.\nTable of Contents\nWhat is Tokenization?\nThe Tokenization Pipeline\nTokenization Algorithms\nAccessing\ntokenizers\nthrough\ntransformers\nThe Tokenizer Class Hierarchy in\ntransformers\nAutoTokenizer\nAutomatically Selects the Correct Tokenizer Class\nv5 Separates Tokenizer Architecture from Trained Vocab\nSummary\nFor experts: If you're already familiar with the concepts and want to understand the changes in v5, go to\nv5 Separates Tokenizer Architecture from Trained Vocab\nBefore diving into the changes, let's quickly cover what tokenization does and how the pieces fit together.\nWhat is tokenization?\nLanguage models don't read raw text. They consume sequences of integers usually called\ntoken IDs or input IDs\n. Tokenization is the process of converting raw text into these token IDs. (Try the tokenization playground\nhere\nto visualize tokenization.)\nTokenization is a broad concept used across natural language processing and text processing generally. This post focuses specifically on tokenization for Large Language Models (LLMs) using the\ntransformers\nand\ntokenizers\nlibraries.\nfrom\ntransformers\nimport\nAutoTokenizer\ntokenizer = AutoTokenizer.from_pretrained(\n\"HuggingFaceTB/SmolLM3-3B\"\n)\n\ntext =\n\"Hello world\"\ntokens = tokenizer(text)\nprint\n(tokens[\n\"input_ids\"\n])\n# [9906, 1917]\nprint\n(tokenizer.convert_ids_to_tokens(tokens[\n\"input_ids\"\n]))\n# ['Hello', 'Ġworld']\nĠworld\n(above) is a single token that represents the character sequence \" world\" (with the space).\nA\ntoken\nis the smallest string unit the model sees. It can be a character, word, or subword chunk like \"play\" or \"##ing\" (\"##\" is a pattern, don't worry if you don't completely understand it now 🤗). The\nvocabulary\nmaps each unique token to the token ID.\nfrom\ntransformers\nimport\nAutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n\"HuggingFaceTB/SmolLM3-3B\"\n)\nprint\n(tokenizer.vocab)\n# {'ÎĹÎľ': 106502, 'ĠPeel': 89694, '.languages': 91078, ...}\nA good tokenizer\ncompresses\ntext into the smallest amount of tokens. Fewer tokens means more usable context without increasing model size. Training a tokenizer boils down to finding the best compression rules for your datasets. For example, if you work on Chinese corpus you can sometimes find\nvery nice surprises 😉\n.\nThe tokenization pipeline\nTokenization happens in stages. Each stage transforms text before passing it to the next:\nStage\nPurpose\nExample\nNormalizer\nStandardizes text (lowercasing, unicode normalization, whitespace cleanup)\n\"HELLO World\"\n→\n\"hello world\"\nPre-tokenizer\nSplits text into preliminary chunks\n\"hello world\"\n→\n[\"hello\", \" world\"]\nModel\nApplies the tokenization algorithm (BPE, Unigram, etc.)\n[\"hello\", \" world\"]\n→\n[9906, 1917]\nPost-processor\nAdds special tokens (BOS, EOS, padding)\n[9906, 1917]\n→\n[1, 9906, 1917, 2]\nDecoder\nConverts token IDs back to text\n[9906, 1917]\n→\n\"hello world\"\nEach component is\nindependent\n. You can swap\nnormalizers\nor change the\nalgorithm\nwithout rewriting everything else.\nYou can access the rust based tokenizer through\n_tokenizer\n. We go in more depth about it in\nthis section\nfrom\ntransformers\nimport\nAutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n\"google/gemma-3-270m-it\"\n)\nprint\n(\nf\"\n{tokenizer._tokenizer.normalizer=}\n\"\n)\n# Replace(...)\nprint\n(\nf\"\n{tokenizer._tokenizer.pre_tokenizer=}\n\"\n)\n# Split(...)\nprint\n(\nf\"\n{tokenizer._tokenizer.model=}\n\"\n)\n# BPE(...)\nprint\n(\nf\"\n{tokenizer._tokenizer.post_processor=}\n\"\n)\n# TemplateProcessing(...)\nprint\n(\nf\"\n{tokenizer._tokenizer.decoder=}\n\"\n)\n# Sequence(decoders=[Replace(...), ByteFallback(), Fuse()])\nTokenization algorithms\nThe following algorithms dominate modern language model tokenizers:\nByte Pair Encoding (BPE)\niteratively merges the most frequent character pairs. This algorithm is deterministic and widely used. (Read more about\nBPE\n)\nfrom\ntransformers\nimport\nAutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n\"openai/gpt-oss-20b\"\n)\nprint\n(tokenizer._tokenizer.model)\n# BPE(...)\nUnigram\ntakes a probabilistic approach, selecting the most likely segmentation from a large initial vocabulary. This is more flexible than the strict BPE. (Read more about\nUnigram\n)\nfrom\ntransformers\nimport\nAutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n\"google-t5/t5-base\"\n)\nprint\n(tokenizer._tokenizer.model)\n# Unigram(...)\nWordPiece\nresembles BPE but uses different merge criteria based on likelihood. (Read more about\nWordPiece\n)\nfrom\ntransformers\nimport\nAutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n\"bert-base-uncased\"\n)\nprint\n(tokenizer._tokenizer.model)\n# WordPiece(...)\nAccessing tokenizers through transformers\nThe\ntokenizers\nlibrary is a Rust-based tokenization engine. It is fast, efficient, and completely language model agnostic. The library handles the mechanics of converting text into token IDs and back. The\ntokenizers\nlibrary is a general-purpose tool that implements the tokenization algorithms, but does not implement the conventions that connect those algorithms to specific language models.\nConsider what happens when you use\ntokenizers\ndirectly with the\nSmolLM3-3B\nmodel:\nfrom\ntokenizers\nimport\nTokenizer\n\ntokenizer = Tokenizer.from_pretrained(\n\"HuggingFaceTB/SmolLM3-3B\"\n)\ntext =\n\"Hello world\"\nencodings = tokenizer.encode(text)\nprint\n(encodings.ids)\n# [9906, 1917]\nprint\n(encodings.tokens)\n# ['Hello', 'Ġworld']\nThe output is raw tokenization. You get token IDs and the string pieces they correspond to. Nothing more.\nNow consider what's missing. The\nSmolLM3-3B\nis a\nconversational model\n. When you interact with it, you typically structure your input as a conversation with roles like \"user\" and \"assistant\". The language model expects special formatting tokens to indicate these roles. The raw\ntokenizers\nlibrary has no concept of any of this.\nHow do you bridge the gap between raw tokenization and model requirements?\nThe\ntransformers\nlibrary bridges this gap. The library is primarily known as a model definition library, but it also provides a tokenizer abstraction layer that wraps the raw\ntokenizers\nbackend and adds model-aware functionality.\nHere's the same tokenization with the\ntransformers\nwrapper:\nfrom\ntransformers\nimport\nAutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n\"HuggingFaceTB/SmolLM3-3B\"\n)\n# Format a conversation using the model's chat template\nprompt =\n\"Give me a brief explanation of gravity in simple terms.\"\nmessages = [{\n\"role\"\n:\n\"user\"\n,\n\"content\"\n: prompt}]\ntext = tokenizer.apply_chat_template(\n    messages,\n    tokenize=\nFalse\n,\n    add_generation_prompt=\nTrue\n,\n)\nprint\n(text)\n# <|im_start|>system\n# ...\n# <|im_start|>user\n# Give me a brief explanation of gravity in simple terms.<|im_end|>\n# <|im_start|>assistant\nmodel_inputs = tokenizer([text], add_special_tokens=\nFalse\n, return_tensors=\n\"pt\"\n)\nNotice how the special tokens like\n<|im_start|>\nand\n<|im_end|>\nare applied to the prompt before tokenizing. This is useful for the model to learn where a new sequence starts and ends.\nThe\ntransformers\ntokenizer adds everything the raw library lacks:\nChat template application.\nThe\napply_chat_template\nmethod formats conversations according to the model's expected format, inserting the correct special tokens and delimiters.\nAutomatic special token insertion.\nBeginning-of-sequence and end-of-sequence tokens are added where the model expects them.\nTruncation to context length.\nYou can specify\ntruncation=True\nand the tokenizer will respect the model's maximum sequence length.\nBatch encoding with padding.\nMultiple inputs can be padded to the same length with the correct padding token and direction.\nReturn format options.\nYou can request PyTorch tensors (\nreturn_tensors=\"pt\"\n), NumPy arrays and others.\ntransformers\nimplements the tokenization API that is most commonly used in the entire ML community (\nencode\n,\ndecode\n,\nconvert_tokens_to_ids\n, etc.)\nThe tokenizer class hierarchy in transformers\nThe\ntransformers\nlibrary organizes tokenizers into a class hierarchy. At the top sits a base class that defines the common interface. Below it, backend classes handle the actual tokenization using different engines. At the bottom, model-specific classes configure the backends for particular models.\nThe class hierarchy for tokenizers inside transformers\nPreTrainedTokenizerBase\ndefines the common interface for all tokenizers\nPreTrainedTokenizerBase\nis the abstract base class for all tokenizers in\ntransformers\n. It defines the interface that every tokenizer must implement.\nThe base class handles functionality that doesn't depend on the tokenization backend:\nSpecial token properties.\nProperties like\nbos_token\n,\neos_token\n,\npad_token\n, and\nunk_token\nare defined here. These properties provide access to the special tokens that models use to mark sequence boundaries and handle unknown inputs.\nEncoding interface.\nThe\n__call__\nmethod,\nencode\n, and\nencode_plus\nmethods are defined here. These methods accept text input and return token IDs along with attention masks and other metadata.\nDecoding interface.\nThe\ndecode\nand\nbatch_decode\nmethods convert token IDs back to text.\nSerialization.\nThe\nsave_pretrained\nand\nfrom_pretrained\nmethods handle downloading the correct files, reading information, saving tokenizers to disk etc.\nChat template support.\nThe\napply_chat_template\nmethod lives here, formatting conversations according to Jinja templates stored in the tokenizer configuration.\nEvery tokenizer in\ntransformers\nultimately inherits from\nPreTrainedTokenizerBase\n. The base class ensures consistent behavior across all tokenizers, regardless of which backend they use for the actual tokenization.\nTokenizersBackend\nwraps the\ntokenizers\nlibrary\nTokenizersBackend\nis the primary backend class for most modern tokenizers. It inherits from\nPreTrainedTokenizerBase\nand wraps the Rust-based\ntokenizers\nlibrary.\nThe class stores the Rust tokenizer object internally:\nclass\nTokenizersBackend\n(\nPreTrainedTokenizerBase\n):\ndef\n__init__\n(\nself, tokenizer_object, ...\n):\n        self._tokenizer = tokenizer_object\n# The Rust tokenizer\n...\nWhen you call encoding methods on a\nTokenizersBackend\ntokenizer, the class delegates the actual tokenization to the Rust backend:\ndef\n_batch_encode_plus\n(\nself, batch_text_or_text_pairs, ...\n):\n    encodings = self._tokenizer.encode_batch(batch_text_or_text_pairs, ...)\n    ...\nThe Rust backend performs computationally intensive work, while the Python wrapper adds the model-aware features on top.\nMany model-specific tokenizers inherit from\nTokenizersBackend\n, examples include:\nLlamaTokenizer\nGemmaTokenizer\nThese model-specific classes configure the backend with the correct vocabulary, merge rules, special tokens, and normalization settings for their respective models.\nPythonBackend\nprovides a pure-Python mixin\nPythonBackend\ninherits from\nPreTrainedTokenizerBase\nand implements tokenization in pure Python. The class is aliased as\nPreTrainedTokenizer\n.\nThe pure-Python backend exists for several reasons:\nCustom tokenization logic.\nSome models require tokenization behavior that doesn't fit the standard\ntokenizers\npipeline.\nLegacy compatibility.\nOlder model implementations may rely on Python-specific behavior.\nThe Python backend is slower than the Rust backend. For most use cases, the Rust-backed\nTokenizersBackend\nis preferred.\nModel-specific tokenizers that inherit from\nPythonBackend\n(or its alias\nPreTrainedTokenizer\n) include some older or specialized models, like:\nCTRLTokenizer\nCanineTokenizer\nSentencePieceBackend\nhandles SentencePiece models\nSentencePieceBackend\ninherits from\nPythonBackend\nand provides integration with Google's\nSentencePiece\nlibrary. SentencePiece is a standalone tokenization library that many models use, particularly those trained by Google.\nThe backend wraps a SentencePiece processor:\nclass\nSentencePieceBackend\n(\nPythonBackend\n):\ndef\n__init__\n(\nself, vocab_file, ...\n):\n        self.sp_model = spm.SentencePieceProcessor()\n        self.sp_model.Load(vocab_file)\n        ...\nModels that use SentencePiece tokenization inherit from this backend. Examples include:\nSiglipTokenizer\nBartphoTokenizer\nThe SentencePiece backend inherits from\nPythonBackend\nrather than directly from\nPreTrainedTokenizerBase\nbecause it shares much of the same interface and padding/truncation logic.\nAutoTokenizer automatically selects the correct tokenizer class\nAutoTokenizer\nis the recommended entry point for loading tokenizers. It automatically determines which tokenizer class to use for a given model and returns an instance of that class.\nfrom\ntransformers\nimport\nAutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\n\"gpt2\"\n)\nBehind the scenes,\nAutoTokenizer\nperforms these steps:\nDownload the tokenizer configuration.\nThe\nfrom_pretrained\nmethod fetches\ntokenizer_config.json\nfrom the Hub (or from a local directory).\nIdentify the model type.\nThe configuration contains metadata that\nidentifies the model type\n(e.g., \"gpt2\", \"llama\", \"bert\").\nLook up the tokenizer class.\nAutoTokenizer\nmaintains a mapping called\nTOKENIZER_MAPPING_NAMES\nthat maps model types to tokenizer class names:\nTOKENIZER_MAPPING_NAMES = {\n\"gpt2\"\n:\n\"GPT2Tokenizer\"\n,\n\"llama\"\n:\n\"LlamaTokenizer\"\n,\n\"bert\"\n:\n\"BertTokenizer\"\n,\n    ...\n}\nInstantiate the correct class.\nAutoTokenizer\nimports the appropriate tokenizer class and calls its\nfrom_pretrained\nmethod.\nReturn the configured tokenizer.\nYou receive a fully configured, model-specific tokenizer ready for use.\nThe benefit of\nAutoTokenizer\nis that you don't need to know which tokenizer class a model uses. Whether a model uses\nLlamaTokenizer\n,\nGPT2Tokenizer\n, or\nBertTokenizer\n, the same\nAutoTokenizer.from_pretrained(\"model-name\")\ncall works.\nThe tokenizer system in\ntransformers\nforms a layered architecture:\nLayer\nComponent\nResponsibility\nEntry Point\nAutoTokenizer\nAutomatically selects and instantiates the correct tokenizer class\nModel-Specific\nLlamaTokenizer\n,\nGPT2Tokenizer\n, etc.\nConfigures the backend with model-specific architecture of normalizer, pre tokenizer, etc, special tokens, and settings\nBackend\nTokenizersBackend\n,\nPythonBackend\n,\nSentencePieceBackend\nImplements the actual tokenization using a specific engine\nBase\nPreTrainedTokenizerBase\nDefines the common interface and shared functionality\nEngine\ntokenizers\n(Rust), SentencePiece, Pure Python\nPerforms raw tokenization\nv5 Separates Tokenizer Architecture from Trained Vocab\nThe most significant change in Transformers v5 is a philosophical shift in how tokenizers are defined.\nTokenizers now work like PyTorch's\nnn.Module\n: you define the architecture first, then fill it with learned parameters.\nThe problem with v4: tokenizers were opaque and tightly coupled\nIn v4, tokenizers were black boxes tied to pretrained checkpoint files. If you loaded\nLlamaTokenizerFast\n, you couldn't easily answer basic questions about it:\nIs it BPE or Unigram?\nHow does it normalize text?\nWhat pre-tokenization strategy does it use?\nWhat are the special tokens and their positions?\nThe\n__init__\nmethod gave no clues. You had to dig through serialized files or external documentation to understand what the tokenizer actually did.\nLlamaTokenizerFast\nas seen in v4\ntransformers\nv4 also maintained two parallel implementations for every model:\na \"slow\" Python tokenizer (\nLlamaTokenizer\ninheriting from\nPreTrainedTokenizer\n) and\na \"fast\" Rust-backed tokenizer (\nLlamaTokenizerFast\ninheriting from\nPreTrainedTokenizerFast\n).\nThis meant:\nTwo files per model\n(e.g.,\ntokenization_llama.py\nand\ntokenization_llama_fast.py\n)\nCode duplication\nacross hundreds of models\nBehavioral discrepancies\nbetween slow and fast versions, leading to subtle bugs\nA growing test suite\ndedicated to verifying that slow and fast tokenizers produced identical outputs\nUser confusion\nabout which tokenizer to use and when\nWorst of all, you couldn't create an empty tokenizer architecture. If you wanted to train a LLaMA-style tokenizer on your own data, there was no clean way to instantiate a \"blank\" LLaMA tokenizer and fill it with your vocabulary and merges. Tokenizers existed only as loaded checkpoints, not as configurable templates.\nThe v5 solution: architecture and parameters are now separate\nv5 treats tokenizer architecture (normalizer, pre-tokenizer, model type, post-processor, decoder) as distinct from trained parameters (vocabulary, merges). This mirrors how PyTorch separates model architecture from learned weights.\nWith\nnn.Module\n, you define layers first:\nfrom\ntorch\nimport\nnn\n\nmodel = nn.Sequential(\n    nn.Embedding(vocab_size, embed_dim),\n    nn.Linear(embed_dim, hidden_dim),\n)\n# Architecture defined; weights initialized randomly or loaded later\nV5 tokenizers follow the same pattern:\nfrom\ntransformers\nimport\nLlamaTokenizer\n# Instantiate the architecture\ntokenizer = LlamaTokenizer()\n# Train on your own data to fill in vocab and merges\ntokenizer.train(files=[\n\"my_corpus.txt\"\n])\nThe tokenizer class now explicitly declares its structure. Looking at\nLlamaTokenizer\nin v5, you can immediately see:\nIt uses\nBPE\nas its tokenization model\nIt may add a\nprefix space\nbefore text\nIts special tokens (\nunk\n,\nbos\n,\neos\n) sit at specific vocabulary positions\nIt does\nnot normalize\ninput text\nIts decoder\nreplaces the metaspace character\n▁\nwith spaces\nLlamaTokenizer\nas seen in v5\ntransformers\nThis transparency was impossible in v4, where the same information was buried in serialized files.\nOne file, one backend, one recommended path\nv5 consolidates the two-file system\ninto a single file per model\n.\nLlamaTokenizer\nnow inherits from\nTokenizersBackend\n, which wraps the Rust-based tokenizer that was previously exposed as the “fast” implementation and is now the default.\nThe former “slow” Python implementation lives explicitly behind\nPythonBackend\n, and\nSentencePieceBackend\nremains for models that require it, but\nRust-backed tokenization is the preferred default\n.\nThis change eliminates:\nDuplicate code across slow/fast implementations\nThe confusing\nTokenizer\nvs\nTokenizerFast\nnaming convention\nTest suites dedicated to checking slow-fast parity\nUsers now have one clear entry point. Advanced users who need to customize can still access lower-level components, but the library no longer forces everyone to navigate two parallel implementations.\nYou can now train model specific tokenizers from scratch\nSuppose you want a tokenizer that behaves exactly like LLaMA's – same normalization, same pre-tokenization, same BPE model type – but trained on a domain-specific corpus (medical text, legal documents, a new language). In v4, this required manually reconstructing the tokenizer pipeline from low-level\ntokenizers\nlibrary primitives. In v5, you can instantiate the architecture directly and call\ntrain\n:\nfrom\ntransformers\nimport\nLlamaTokenizer\nfrom\ndatasets\nimport\nload_dataset\n# Initialize blank tokenizer\ntokenizer = LlamaTokenizer()\n\ndataset = load_dataset(\n\"wikitext\"\n,\n\"wikitext-2-raw-v1\"\n, split=\n\"train\"\n)\ndef\nget_training_corpus\n():\n    batch =\n1000\nfor\ni\nin\nrange\n(\n0\n,\nlen\n(dataset), batch):\nyield\ndataset[i : i + batch][\n\"text\"\n]\n\ntrained_tokenizer = tokenizer.train_new_from_iterator(\n    text_iterator=get_training_corpus(),\n    vocab_size=\n32000\n,\n    length=\nlen\n(dataset),\n    show_progress=\nTrue\n,\n)\n\ntrained_tokenizer.push_to_hub(\n\"my_custom_tokenizer\"\n)\n\ntokenizer = LlamaTokenizer.from_pretrained(\n\"my_custom_tokenizer\"\n)\nThe resulting tokenizer will have your custom vocabulary and merge rules, but will process text identically to how a standard LLaMA tokenizer would with the same whitespace handling, same special token conventions, same decoding behavior.\nAspect\nV4\nV5\nFiles per model\nTwo (\ntokenization_X.py\n,\ntokenization_X_fast.py\n)\nOne (\ntokenization_X.py\n)\nDefault backend\nSplit between Python and Rust\nRust (\nTokenizersBackend\n) preferred\nArchitecture visibility\nHidden in serialized files\nExplicit in class definition\nTraining from scratch\nRequired manual pipeline construction\ntokenizer.train(files=[...])\nComponent inspection\nDifficult, undocumented\nDirect properties (\ntokenizer.normalizer\n, etc.)\nParent classes\nPreTrainedTokenizer\n,\nPreTrainedTokenizerFast\nTokenizersBackend\n(or\nSentencePieceBackend\n,\nPythonBackend\n)\nThe shift from \"tokenizers as loaded checkpoints\" to \"tokenizers as configurable architectures\" makes the library more modular, more transparent, and more aligned with how practitioners think about building ML systems.\nSummary\nTransformers v5 brings three improvements to tokenization:\nOne file per model\ninstead of separate slow/fast implementations\nVisible architecture\nso you can inspect normalizers, pre-tokenizers, and decoders\nTrainable templates\nthat let you create custom tokenizers matching any model's design\nThe wrapper layer between\ntokenizers\nand Transformers remains essential. It adds model awareness, context lengths, chat templates, special tokens, that raw tokenization doesn't provide. V5 just makes that layer clearer and more customizable.\nIf you are looking to learn more about tokenization here are some resources:\nLet's build the GPT Tokenizer\nGotchas in Tokenizer Behavior Every Developer Should Know\nChat Templates\nA list of resources we have gathered from the community!\nModels mentioned in this article\n2\nSpaces mentioned in this article\n1\nMore Articles from our Blog\ntokenizers\ntokenization\nrust\ntokenizers v1: encode, decode and scaling, measured\n91\nSeptember 21, 2026\nannouncement\nmultimodal\ntransformers\nWelcome Inkling by Thinking Machines\n+1\n167\nJuly 15, 2026\nCommunity\nFHSEOHub\nDec 20, 2025\nI think it depends on the nature of tools\nSee translation\n1 reply\n·\n👀\n1\n1\n+\nariG23498\nArticle author\nDec 23, 2025\nWhat depends on the nature of tools? 😳\nSee translation\ndeleted\nDec 21, 2025\n•\nThis comment has been hidden\nSifal\nJan 10\nThanks for doing this! I had to train some tokenizers with the v4, it was indeed not straightforward to understand the behavior.\nI had two questions:\nYou said: older model implementations may rely on Python-specific behavior.\nCurious if you had any example\nYou sometimes say \"fast\" (between quotes) is it just to refer to the fastTokenizers backend or can the implementation actually be slower than the python implementation because of some kind of rust overhead?\nSee translation\n2 replies\n·\nariG23498\nArticle author\nJan 12\nGlad that this was useful to you.\nAll the classes that extend the\nPreTrainedTokenizer\n(which is an alias to the\nPythonBackend\nwill serve as you examples. (\nGitHub Search\n)\nThe rust backend is faster compare to the other implementations.\nSee translation\n🤗\n1\n1\n+\nExpand 1\n\t\t\t\t\t\treply\nEdit\nPreview\nUpload images, audio, and videos by dragging in the text input, pasting, or\nclicking here\n.\nTap or paste here to upload images\nComment\n·\nSign up\nor\nlog in\nto comment\nUpvote\n128\n+116\nModels mentioned in this article\n2\nSpaces mentioned in this article\n1",
  "metadata": {
    "title": "Tokenization in Transformers v5: Simpler, Clearer, and More Modular",
    "description": "We’re on a journey to advance and democratize artificial intelligence through open source and open science.",
    "keywords": null,
    "author": null,
    "og:title": "Tokenization in Transformers v5: Simpler, Clearer, and More Modular",
    "og:description": "We’re on a journey to advance and democratize artificial intelligence through open source and open science.",
    "og:type": "website",
    "og:url": "https://huggingface.co/blog/tokenizers",
    "og:image": "https://huggingface.co/blog/assets/tokenizers/thumbnail.png",
    "twitter:card": "summary_large_image",
    "twitter:site": "@huggingface",
    "twitter:image": "https://huggingface.co/blog/assets/tokenizers/thumbnail.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/c0bb8359279a/code/body-000.txt",
      "chars": 282,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-001.txt",
      "chars": 195,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-002.txt",
      "chars": 462,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-003.txt",
      "chars": 150,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-004.txt",
      "chars": 153,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-005.txt",
      "chars": 155,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-006.txt",
      "chars": 239,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-007.txt",
      "chars": 622,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-008.txt",
      "chars": 174,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-009.txt",
      "chars": 145,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-010.txt",
      "chars": 188,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-011.txt",
      "chars": 90,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-012.txt",
      "chars": 127,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-013.txt",
      "chars": 195,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-014.txt",
      "chars": 196,
      "language": "text"
    },
    {
      "file": "attachments/c0bb8359279a/code/body-015.txt",
      "chars": 630,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/0102bcf0db4e822b8bf61ae92305680f.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-2f7e744eea9dc020a91d.svg",
      "src": "https://huggingface.co/avatars/0102bcf0db4e822b8bf61ae92305680f.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/af82e08ae137c80008bc9a020182da3f.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-d5d8d630e88a256696da.svg",
      "src": "https://huggingface.co/avatars/af82e08ae137c80008bc9a020182da3f.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/15720cf8cd69a7fecfc749f5524c2500.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-0ae82b336f7440b05d20.svg",
      "src": "https://huggingface.co/avatars/15720cf8cd69a7fecfc749f5524c2500.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/c5f40dde9d29e771aad5e7e76fb6ce22.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-dd07075ee52de032ac1c.svg",
      "src": "https://huggingface.co/avatars/c5f40dde9d29e771aad5e7e76fb6ce22.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/6317233cc92fd6fee317e030/cJHSvvimr1kqgQfHOjO5n.png",
      "bytes": 5138,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-4846b3d76d1e92e2d185.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/6317233cc92fd6fee317e030/cJHSvvimr1kqgQfHOjO5n.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/42ZnpUdf-jANafNZ_2-Qg.png",
      "bytes": 11198,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-ccc5474e7c100b5e5ed3.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/42ZnpUdf-jANafNZ_2-Qg.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/thumbnail.png",
      "bytes": 85880,
      "reused": true,
      "local_path": "attachments/c0bb8359279a/images/006.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/thumbnail.png",
      "alt": "thumbnail",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/hierarchy.png",
      "bytes": 757388,
      "reused": true,
      "local_path": "attachments/c0bb8359279a/images/007.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/hierarchy.png",
      "alt": "class hierarchy",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/v4-llama.png",
      "bytes": 171196,
      "reused": true,
      "local_path": "attachments/c0bb8359279a/images/008.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/v4-llama.png",
      "alt": "v4 llama",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/v5-llama.png",
      "bytes": 176657,
      "reused": true,
      "local_path": "attachments/c0bb8359279a/images/009.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/v5-llama.png",
      "alt": "v5 llama",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/blog/assets/tokenizers-v1/thumbnail.png",
      "bytes": 353084,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-e0ea47069fb9008688bd.png",
      "src": "https://huggingface.co/blog/assets/tokenizers-v1/thumbnail.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1674683851722-62441cb7456803e95009a08f.jpeg",
      "bytes": 17020,
      "reused": true,
      "local_path": "attachments/c0bb8359279a/images/002.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1674683851722-62441cb7456803e95009a08f.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1608146735109-5fcfb7c407408029ba3577e2.png",
      "bytes": 7846,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-4a04daefaa5f674778a1.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1608146735109-5fcfb7c407408029ba3577e2.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1666977434736-617bc8d1000dbbbf7c225eed.png",
      "bytes": 24660,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-c3b687f6faba7d8a43c4.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1666977434736-617bc8d1000dbbbf7c225eed.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/5e3aec01f55e2b62848a5217/PMKS0NNB4MJQlTSFzh918.jpeg",
      "bytes": 11598,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-741cc42f07a702f49510.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/5e3aec01f55e2b62848a5217/PMKS0NNB4MJQlTSFzh918.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/blog/assets/thinkingmachines-inkling/thumbnail.png",
      "bytes": 137478,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-b7dfa2e2f2824d30ffa1.png",
      "src": "https://huggingface.co/blog/assets/thinkingmachines-inkling/thumbnail.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/62d648291fa3e4e7ae3fa6e8/oatOwf8Xqe5eDbCSuYqCd.png",
      "bytes": 4308,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-2f7cd9aaae2da43f9642.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/62d648291fa3e4e7ae3fa6e8/oatOwf8Xqe5eDbCSuYqCd.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/6141a88b3a0ec78603c9e784/DJsxSmWV39M33JFheLobC.jpeg",
      "bytes": 15094,
      "reused": true,
      "local_path": "attachments/c0bb8359279a/images/004.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/6141a88b3a0ec78603c9e784/DJsxSmWV39M33JFheLobC.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1617264212503-603d25b75f9d390ab190b777.jpeg",
      "bytes": 9492,
      "reused": true,
      "local_path": "attachments/c0bb8359279a/images/005.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1617264212503-603d25b75f9d390ab190b777.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/608aabf24955d2bfc3cd99c6/-YxmtpzEmf3NKOTktODRP.jpeg",
      "bytes": 9712,
      "reused": true,
      "local_path": "attachments/c0bb8359279a/images/001.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/608aabf24955d2bfc3cd99c6/-YxmtpzEmf3NKOTktODRP.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/T8RHDXgCMvpizznuCKDbl.jpeg",
      "bytes": 3724,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-b4871fba38f2eebd5222.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/T8RHDXgCMvpizznuCKDbl.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/647431447ff32a81ac70fb71/wweMKErGYcyqd9u1sDd7C.jpeg",
      "bytes": 7632,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-8957863f8d7f7eebd15d.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/647431447ff32a81ac70fb71/wweMKErGYcyqd9u1sDd7C.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/b84d1fd6306cd4958c6c924c3dadf371.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-84977497f212a4dcef6d.svg",
      "src": "https://huggingface.co/avatars/b84d1fd6306cd4958c6c924c3dadf371.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/oxdEavFaRybUF68NOKSvN.png",
      "bytes": 9260,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-f38c3719e8de0bb1d1a0.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/oxdEavFaRybUF68NOKSvN.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/63107b18e87051f3e3e0f598/R9onir4Y0MZuq1jEWCZ2-.jpeg",
      "bytes": 8014,
      "reused": false,
      "local_path": "attachments/c0bb8359279a/images/body-e4387ae3690d3ed5a2f6.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/63107b18e87051f3e3e0f598/R9onir4Y0MZuq1jEWCZ2-.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/61929226ded356549e20c5da/ONUjP2S5fUWd07BiFXm0i.jpeg",
      "bytes": 3442,
      "reused": true,
      "local_path": "attachments/c0bb8359279a/images/003.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/61929226ded356549e20c5da/ONUjP2S5fUWd07BiFXm0i.jpeg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=zduSFxRajkE",
      "src": "https://youtu.be/zduSFxRajkE?si=ZAfCjZjpyPHsnyfF",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/c0bb8359279a/videos/repair-zduSFxRajkE.mp4",
      "bytes": 389606169,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "8014.241000",
      "reused": true,
      "reused_from": "xinxing-zimeiti-corpus/attachments/36db3bbf84a3/videos/zduSFxRajkE.mp4"
    }
  ],
  "links": [
    {
      "href": "https://huggingface.co/blog",
      "label": "Back to Articles",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/blog/blob/main/tokenizers.md",
      "label": "Update on GitHub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/login?next=%2Fblog%2Ftokenizers",
      "label": "Upvote 128",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/bitmman-nch",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/harshkhokhariya",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/hiepnh",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/parthik1",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/tomaarsen",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/physmatician",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/itazap",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/ariG23498",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/ArthurZ",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/sergiopaniego",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/merve",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/pcuenq",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/thumbnail.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/transformers-v5",
      "label": "Transformers v5",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers/pull/40936/files",
      "label": "big tokenizers reformat",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/spaces/Xenova/the-tokenizer-playground",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers",
      "label": "transformers",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/tokenizers",
      "label": "tokenizers",
      "type": "a"
    },
    {
      "href": "https://x.com/suchenzang/status/1697862650053660721",
      "label": "very nice surprises 😉",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/tokenizers/en/api/normalizers",
      "label": "normalizers",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/tokenizers/en/api/models",
      "label": "algorithm",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/learn/llm-course/en/chapter6/5",
      "label": "BPE",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/learn/llm-course/en/chapter6/7",
      "label": "Unigram",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/learn/llm-course/en/chapter6/6",
      "label": "WordPiece",
      "type": "a"
    },
    {
      "href": "http://hf.co/HuggingFaceTB/SmolLM3-3B",
      "label": "SmolLM3-3B",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/hierarchy.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_utils_base.py",
      "label": "PreTrainedTokenizerBase",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_utils_tokenizers.py",
      "label": "TokenizersBackend",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_python.py",
      "label": "PythonBackend",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_utils_sentencepiece.py",
      "label": "SentencePieceBackend",
      "type": "a"
    },
    {
      "href": "https://github.com/google/sentencepiece",
      "label": "SentencePiece",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/models/auto/tokenization_auto.py",
      "label": "AutoTokenizer",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/openai-community/gpt2/blob/main/config.json",
      "label": "identifies the model type",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/v4-llama.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers/blob/0a8465420eecbac1c6d7dd9f45c08dd96b8c5027/src/transformers/models/llama/tokenization_llama.py",
      "label": "It uses BPE",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/v5-llama.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://youtu.be/zduSFxRajkE?si=ZAfCjZjpyPHsnyfF",
      "label": "Let's build the GPT Tokenizer",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/qgallouedec/gotchas-in-tokenizer-behavior",
      "label": "Gotchas in Tokenizer Behavior Every Developer Should Know",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/chat-templates",
      "label": "Chat Templates",
      "type": "a"
    },
    {
      "href": "https://x.com/ariG23498/status/1999058214906888237",
      "label": "A list of resources we have gathered from the community!",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/tokenizers-v1",
      "label": "tokenizers tokenization rust tokenizers v1: encode, decode and scaling, measured 91 September 21, 2026",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/thinkingmachines-inkling",
      "label": "announcement multimodal transformers Welcome Inkling by Thinking Machines +1 167 July 15, 2026",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/FHSEOHub",
      "label": "FHSEOHub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/Sifal",
      "label": "Sifal",
      "type": "a"
    },
    {
      "href": "https://github.com/search?q=repo%3Ahuggingface%2Ftransformers%20PreTrainedTokenizer&type=code",
      "label": "GitHub Search",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/join?next=%2Fblog%2Ftokenizers",
      "label": "Sign up",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/jeanjacquesfoucault",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/serelk",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/ucyang",
      "label": "",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:26Z",
  "source_url": "https://huggingface.co/blog/tokenizers",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 23619,
  "expected_body_images": 26,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/c0bb8359279a/pages/source.html",
  "local_html": "attachments/c0bb8359279a/pages/article.html",
  "body_base_path": "attachments/c0bb8359279a/pages/",
  "body_markdown_path": "attachments/c0bb8359279a/pages/article.md"
}