{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Seth Wheeler",
  "title_listed": "Was TinyStories the Domain or the Vocabulary?",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/megapixel99/was-tinystories-the-domain-or-the-vocabulary-2gg0",
  "publish_date": "2026-09-10",
  "corpus_stage": "",
  "notes": "",
  "id": "b3c34118259f",
  "collected_at": "2026-10-02T13:51:55",
  "title": "Was TinyStories the Domain or the Vocabulary? - DEV Community",
  "body_markdown": "I have been running a long series of experiments on whether small models can be made to behave usefully, and one obvious thing to borrow is whatever TinyStories did. That paper trained very small transformers to write coherent children's stories, and the detail everyone remembers is the vocabulary: roughly 1,500 words.\nAn earlier experiment of mine had already varied the\ndomain\nat a fixed vocabulary, restricting a corpus to a single arXiv category, and found a real win: +3.7 points of top-1, perplexity down 1.8x, and 17x less degeneration into repetition. So the natural follow-up is the other axis. Hold the domain and vary the vocabulary. If the vocabulary is the lever, capping it should help.\nIt does not help; it hurts, by 4 points, and it wrecks the generated text. The research repo is not public, so the figures below come from the experiment's own harness and its recorded\nresults.json\nrather than from anything you can clone.\nThe fairness problem, first\nA smaller vocabulary makes next-token prediction trivially easier, because everything outside the cap becomes\n<unk>\nand\n<unk>\nbecomes a frequent, easy target. Any arm with a smaller vocabulary will look better on a naive score, and it will look better for a reason that has nothing to do with capability.\nSo every arm is scored only on positions whose target is inside the\nsmallest\nvocabulary, which is 1,498 types. Identical positions, identical targets, across all three arms;\n<unk>\nis never a scored target. 3,000 scored positions each.\nGetting that right was most of the work, and it is the part I would want someone to check first. Without it the headline result reverses, and the reversed version is the one that would have confirmed what I expected to find.\nThe numbers\nSame corpus (846K training tokens of cs.CL abstracts), same model at 4 layers and d=256 with a 128-token context, same budget per arm. Vocabulary capped at 1,500, 4,000 and 8,000 types.\nvocab cap\ntraining\n<unk>\nrate\ntop-1\ntop-5\nMRR\nperplexity\n1,500\n18.5%\n0.211\n0.446\n0.321\n75.9\n4,000\n7.5%\n0.251\n0.460\n0.349\n72.9\n8,000\n3.2%\n0.253\n0.454\n0.348\n86.4\nCap 1,500 is 4 points behind on identical scored positions. There is no accuracy to buy here at all, which was not the result I was expecting to write down.\nThe generation metrics are worse than the accuracy ones:\nvocab cap\nrepeated 4-gram\ndistinct-4\nnovel 4-gram\n<unk>\nin output\n1,500\n0.075\n0.925\n0.291\n36.8%\n4,000\n0.050\n0.950\n0.557\n15.0%\n8,000\n0.044\n0.956\n0.671\n1.8%\nMore than a third of the emitted tokens at cap 1,500 are the unknown-word placeholder. Novel 4-grams collapse from 0.671 to 0.291 while repetition rises. The model becomes\nmore\ncopy-like, which is precisely the opposite of the effect I was trying to reproduce.\nThe samples make it concrete. At cap 1,500:\nwe propose\n<unk>\n- guided\n<unk>\n, a multimodal adaptive multimodal\n<unk>\nframework for\n<unk>\nmultimodal\n<unk>\nand\n<unk>\n<unk>\nanalysis .\nAt cap 8,000:\nwe propose\n<unk>\n, a framework that combines long - horizon memory with a lightweight context model with adaptive memory . in this architecture , the model observes the user before the tool to a small fraction of the context …\nThe second one is not good prose, and it is recognisably prose. The first is a grammatical skeleton with every content word deleted.\nWhy capping a vocabulary is not simplifying\nThat last observation is the whole finding, and it took the samples rather than the metrics to see it.\nTinyStories restricted the\ngenerative process\n. The stories were\nwritten\nusing simple words, so every token in the corpus carries meaning, and a small model learning that corpus is learning a language in which short words are sufficient. Capping the vocabulary of hard text restricts only the\nrepresentation\n. The content words are not simplified, they are deleted and replaced with a placeholder that means nothing, and what remains is grammar with nothing to say.\nSo the two operations look similar written down as \"small vocabulary\" and are opposites in what they do to the data. One produces a corpus that is genuinely simpler; the other produces a corpus that is damaged, and the damage falls entirely on the words that carry the information.\nYou cannot recover TinyStories' benefit post-hoc from difficult text. If you want that effect on your own corpus, you need text that was actually written simply, not a smaller softmax over text that was not.\nThe part that is still useful\nThere is a mild sweet spot at 4,000 types: the best perplexity of the three at 72.9, and a top-1 statistically tied with 8,000 at four times fewer types. Worth knowing if you are fitting a model into a footprint, because the embedding table scales with the vocabulary and that table is usually most of a small model's parameters.\nBut that is a memory optimisation and not a capability one, and the two get conflated constantly because they are the same dial. Turning it down saves bytes; it does not make the model better at anything.\nWhat generalises\nThe reusable part is not about vocabularies. It is that \"restrict X\" can name two operations that are opposites, depending on whether you restrict the process that generated the data or the encoding of data already generated. Domain restriction in the earlier experiment was the first kind: the corpus really was drawn from a narrower distribution. Vocabulary capping here was the second kind, and it read as the same sort of intervention right up until the samples were printed.\nThe other reusable part is the scoring fix. When one arm of an experiment makes the task easier, the arm that wins is the one that made it easiest, and no amount of care elsewhere recovers that. Restricting the scored positions to the intersection cost about ten lines and it is the only reason these numbers mean anything.",
  "metadata": {
    "title": "Was TinyStories the Domain or the Vocabulary? - DEV Community",
    "description": "TinyStories is famous for a ~1,500-word vocabulary, so I capped mine at 1,500 and scored every arm on identical positions. It came out 4 points worse at top-1 than a 4,000-word cap, and more than a third of its output was the unknown-word token. The transferable lever was the domain, not the vocabulary. Tagged with ai, machinelearning, llm, nlp.",
    "keywords": "ai, machinelearning, llm, nlp, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/megapixel99/was-tinystories-the-domain-or-the-vocabulary-2gg0",
    "og:title": "Was TinyStories the Domain or the Vocabulary?",
    "og:description": "TinyStories is famous for a ~1,500-word vocabulary, so I capped mine at 1,500 and scored every arm on identical positions. It came out 4 points worse at top-1 than a 4,000-word cap, and more than a third of its output was the unknown-word token. The transferable lever was the domain, not the vocabulary.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fiah631umkmyre5emi06w.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Was TinyStories the Domain or the Vocabulary?",
    "twitter:description": "TinyStories is famous for a ~1,500-word vocabulary, so I capped mine at 1,500 and scored every arm on identical positions. It came out 4 points worse at top-1 than a 4,000-word cap, and more than a third of its output was the unknown-word token. The transferable lever was the domain, not the vocabulary.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fiah631umkmyre5emi06w.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:40Z",
  "source_url": "https://dev.to/megapixel99/was-tinystories-the-domain-or-the-vocabulary-2gg0",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 5766,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/b3c34118259f/pages/source.html",
  "local_html": "attachments/b3c34118259f/pages/article.html",
  "body_base_path": "attachments/b3c34118259f/pages/",
  "body_markdown_path": "attachments/b3c34118259f/pages/article.md"
}