{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Seth Wheeler",
  "title_listed": "Just Train More: Measuring the Exchange Rate",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/megapixel99/just-train-more-measuring-the-exchange-rate-1bp0",
  "publish_date": "2026-09-14",
  "corpus_stage": "",
  "notes": "",
  "id": "826382a9780b",
  "collected_at": "2026-10-02T13:51:50",
  "title": "Just Train More: Measuring the Exchange Rate - DEV Community",
  "body_markdown": "The previous experiment\nfound that a count table over the current document overtakes a small transformer at a few hundred tokens of document, and that the transformer's marginal contribution collapses past that point. There is one obvious objection and it deserves a measurement rather than an argument:\nJust train more. 8M tokens is nothing. Of course a 2.45M-parameter transformer loses; train it on a real corpus and the crossover disappears.\nThe objection is directionally right and quantitatively weak, and the useful part is the rate. The research repo is not public, so the figures come from its own harness.\nWhy this needed no new training\nAn earlier experiment had saved checkpoints at 500K, 2M and 8M tokens of code with identical architecture, so the sweep is a re-evaluation rather than a training run. The cache uses no training data at all, which makes its accuracy a constant across the sweep: whatever the transformer gains from 16x more data is exactly what moves the crossover.\nThere is one wrinkle worth stating because it could have quietly broken the comparison. Each checkpoint built its own 16K vocabulary from its own subset, and only 39.8% of the 500K and 8M vocabularies overlap. So every cell is encoded with the vocabulary of the checkpoint it tests, and the corpus count model is built on the same tokens the transformer was trained on. Comparisons are always\nwithin\none data scale; what gets compared across scales is only the crossover position.\nThe table\nIn-distribution held-out code. The value is the paired delta, cache minus transformer, so positive means the zero-parameter cache wins.\ntrain             250      400      600      800     1000     4000\n  500,000        0.073*   0.123*   0.124*   0.193*   0.196*   0.268*\n  2,000,000      0.022    0.077*   0.078*   0.151*   0.135*   0.226*\n  8,000,000     -0.057*   0.006   -0.009    0.055*   0.044*   0.098*\n  (* = paired-significant)\nEnter fullscreen mode\nExit fullscreen mode\nThe cache's own top-1 is 0.311, 0.312, 0.313 at L=250 and 0.487, 0.488, 0.487 at L=4000 across the three rows. It is supposed to be constant, because it never sees the training corpus, and it is, to within a vocabulary difference. That invariance is the experiment's internal control: every bit of movement in the table is the trained component improving, and if the cache row had drifted, nothing else here would be readable.\nThe exchange rate\nA log-linear fit of each row's delta against document length:\ntraining tokens\ncrossover\n500K\n~74 tokens of document\n2M\n~155\n8M\n~492\nSixteen times the training data moved the crossover 6.6x, which is document length times 1.60 per doubling of training data. Extrapolating that rate, which is illustrative rather than measured, since this project's own earlier work is a warning about trusting a slope past its fitted range:\nto be competitive at\nneeds\n1000 tokens of open file\n~23M tokens (3x)\n4000 tokens\n~174M tokens (22x)\n16000 tokens\n~1.3B tokens (167x)\nUsing only the 2M to 8M segment, where the model is less undertrained, the rate is 1.78 per doubling and L=4000 needs about 97M tokens, so 12x. The honest range for matching a zero-parameter cache on a 4000-token file is therefore roughly 10 to 20 times more in-domain code than CPython's standard library and PyTorch combined.\nThat is what makes the objection weak rather than wrong. The crossover does move. It moves at a price, and per-repo completion is precisely the setting where the code to pay it does not exist.\nWhy the rate is so poor: two unrelated slopes\nTraining data lifts the transformer's flat line. It does not give the model access to the document. A 64-token window cannot see a 4000-token file no matter how much text it was trained on, so the transformer's gain from data is roughly the same at every document length: +0.132 at L=250, +0.148 at L=1000, +0.170 at L=4000 across the full 16x.\nThe crossover moves only because the flat line rises, which makes the exchange rate a race between two slopes that have nothing to do with each other. They turn out to be the same size:\n16x more training data moves the transformer from 0.219 to 0.389 at L=4000, so +0.170.\nThe cache seeing 250 rather than 4000 tokens of the file it is already inside moves it from 0.313 to 0.487, so +0.174.\nSixteen times the training corpus buys about as much as letting a zero-parameter mechanism read the rest of the file it is already sitting in. One of those is a data-collection project; the other is already in the editor's buffer.\nA side result that surprised me more than the headline\nAt 8M tokens and L=250 the corpus count table scores 0.415 against the transformer's 0.370. That reproduces an earlier finding, that counting beats the small transformer on code, on a different harness, which is the sort of accidental replication worth keeping.\nCombined with the corpus-plus-cache numbers from a neighbouring experiment, 0.461 rising to 0.551 across these lengths, the transformer does not beat the free components at\nany\ndocument length in this range. It only ever earns its place as a mixture component, at a weight of about 0.2.\nWhat this does not settle, and it is the more serious half\nThe objection had two halves and this addresses one. The other one is that\nparameters and context window are confounded in every experiment in this line.\nEverything here varies training data at a fixed 64-token window, and the cache's advantage is entirely about seeing text the transformer cannot see. A model with a 4096-token window would read the document itself.\nThat is the experiment this line of work still owes, and until it is run, the correct summary is narrow: at a fixed short context, buying document access is far cheaper than buying training data, and the exchange rate is measured rather than argued.\nThree limits on the numbers themselves. The middle checkpoint is the weak link, because the transformer was nearly flat from 500K to 2M and jumped at 8M, so the global fit averages a slow segment with a fast one; both segment rates are given above for that reason. Everything is in-distribution, and cross-project the cache wins earlier, so this is the conservative split for the cache. And the chunking takes text from the start of each file, so larger L also means positions deeper in a file: the within-cell comparisons that make up the whole table are clean, but a single system's trend across L is not.",
  "metadata": {
    "title": "Just Train More: Measuring the Exchange Rate - DEV Community",
    "description": "The obvious objection to a zero-parameter cache beating a small transformer is that the transformer is undertrained. Sixteen times the training data moved the crossover 6.6x, which is document length times 1.60 per doubling: about as much as letting the cache read the rest of the file it was already sitting in. Tagged with ai, machinelearning, llm, datascience.",
    "keywords": "ai, machinelearning, llm, datascience, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/megapixel99/just-train-more-measuring-the-exchange-rate-1bp0",
    "og:title": "Just Train More: Measuring the Exchange Rate",
    "og:description": "The obvious objection to a zero-parameter cache beating a small transformer is that the transformer is undertrained. Sixteen times the training data moved the crossover 6.6x, which is document length times 1.60 per doubling: about as much as letting the cache read the rest of the file it was already sitting in.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fclq45wmkohfq0tbwgkwi.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Just Train More: Measuring the Exchange Rate",
    "twitter:description": "The obvious objection to a zero-parameter cache beating a small transformer is that the transformer is undertrained. Sixteen times the training data moved the crossover 6.6x, which is document length times 1.60 per doubling: about as much as letting the cache read the rest of the file it was already sitting in.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fclq45wmkohfq0tbwgkwi.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/826382a9780b/code/body-000.txt",
      "chars": 303,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://sethwheeler.dev/blog/document-vs-parameters/",
      "label": "The previous experiment",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:40Z",
  "source_url": "https://dev.to/megapixel99/just-train-more-measuring-the-exchange-rate-1bp0",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 6391,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/826382a9780b/pages/source.html",
  "local_html": "attachments/826382a9780b/pages/article.html",
  "body_base_path": "attachments/826382a9780b/pages/",
  "body_markdown_path": "attachments/826382a9780b/pages/article.md"
}