{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Arun Kumar",
  "title_listed": "Speculative decoding won't change your model's distribution. It might still change your output.",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/narotra05hp/speculative-decoding-wont-change-your-models-distribution-it-might-still-change-your-output-3de8",
  "publish_date": "2026-09-06",
  "corpus_stage": "",
  "notes": "",
  "id": "b102a0ad5a0c",
  "collected_at": "2026-10-02T13:54:55",
  "title": "Speculative decoding won't change your model's distribution. It might still change your output. - DEV Community",
  "body_markdown": "There's a thread on the DeepSeek-R1 model page that's been sitting unresolved since March last year, and it bothered me enough to go read the papers.\nSomeone had tried speculative decoding and reported that the output got worse — \"very low quality words for the given context, words it would never generate by itself\". Someone else replied that this is impossible, because the main model verifies and corrects everything the draft model proposes. Neither budged.\nThey're both right. That took me a while to see.\nThe guarantee is real\nThe claim that speculative decoding can't hurt quality isn't marketing. It's proven, twice, independently.\nLeviathan and colleagues at Google introduced the technique in 2022 and state it flatly: \"A strong property of Algorithm 1 is that the output distribution is guaranteed to remain unchanged\". Chen and colleagues at DeepMind arrived at the same place a few months later through a different proof — theirs is \"a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics\".\nSo the person insisting it cannot degrade quality has the papers behind them.\nThe guarantee is also narrower than it sounds\nHere's the part almost every explainer skips, and it's sitting right there in the DeepMind paper:\nbecause the different computation graphs lead to different numerics, we cannot not expect identical outputs\n(The doubled \"not\" is theirs. I left it.)\nRead that against the guarantee and the argument dissolves. The\ndistribution\nis preserved. The\nsample\nis not the same sample. You're drawing from the same distribution through a different computation graph, with different floating-point behaviour in the batched verification pass than in a plain decode.\nSo on any single prompt, you can absolutely get different text. Sometimes worse-looking text. That isn't the method failing — it's what same-distribution-but-different-draw means in practice.\nWhich is exactly what the person reporting bad output experienced. And exactly why the person quoting the theory couldn't reproduce it.\nWorth noting: Leviathan's own abstract does claim \"identical outputs\" for their T5X experiments. That's true for their setup — argmax sampling, one implementation. It's a property of that configuration, not a law.\nIt does more work, not less\nThis is the bit that genuinely surprised me.\nI'd absorbed the idea that speculative decoding is a clever way to do less computation. It's the opposite. Leviathan has an entire section titled\nNumber of Arithmetic Operations\n, and it says the number of concurrent arithmetic operations \"grows by a factor of\" gamma plus one. Every rejected draft token is compute you spent and threw away.\nTotal FLOPs go\nup\n.\nIt's faster anyway, because arithmetic was never what you were waiting on. From the same paper: inference from large models is \"often not bottlenecked on arithmetic operations, but rather on memory bandwidth and communication\". Chen puts it in five words — \"Transformer sampling is typically memory bandwidth bound\".\nYour accelerator is sitting idle waiting on memory. Speculative decoding spends that idle compute to avoid serial steps. It's a trade, and it only pays because one side of the ledger was free.\nThat reframing changed how I think about the whole category. Same shape as FlashAttention, incidentally — a technique everyone describes as making attention cheaper, when it is really about memory movement.\nWhat the numbers actually look like\nChen's Table 1, Chinchilla 70B, batch size 1, K=4:\nResult\nTime per token\nSpeedup\nAutoregressive, XSum\n0.112\n14.1 ms\n1x\nSpeculative, XSum\n0.114\n7.52 ms\n1.92x\nAutoregressive, HumanEval\n45.1%\n14.1 ms\n1x\nSpeculative, HumanEval\n47.0%\n5.73 ms\n2.46x\nTime per token roughly halves. Now look at the quality column — HumanEval goes\nup\n, 45.1 to 47.0.\nDon't read that as an improvement. Elsewhere in the same table, XSum's greedy ROUGE-2 goes down, 0.157 to 0.156. Small movements in both directions across metrics is precisely what a distribution-preserving method should produce. If speculative decoding reliably improved scores, the theory would be wrong.\nWhere the speedup goes away\nBoth papers are more careful than the blog posts about them.\nYou need spare compute — Leviathan is explicit that the algorithm assumes \"we have enough compute resources to support the increased concurrency\". No headroom, no win.\nYou need a high acceptance rate. Every rejected token is wasted work.\nAnd the draft model has to be genuinely cheap. This is the counterintuitive one: against an 11B target, Leviathan found T5-small (77M) beat both T5-base (250M) and T5-large (800M). The best draft model was the smallest one tested, not the most accurate.\nOne more caveat that gets dropped constantly:\nboth papers measured at batch size 1.\nThat's a latency benchmark. Chen notes that other techniques are the ones aimed at throughput \"(at larger batch sizes)\". If you're serving large batches, don't expect these numbers.\nSo who was right in that thread?\nBoth, and the disagreement was never really about speculative decoding.\nThe theory says your distribution is untouched. Your eyes say the text came out different. Both statements are true simultaneously, and the papers are clear about why, and nobody in the thread had read that far.\nIf your output is\nreproducibly\nworse across many prompts, that's an implementation problem — there's a 2025 paper showing some engines break the distribution guarantee outright once batch size exceeds one. If it's one bad generation, that's just a different draw.\nI went through both papers line by line for a longer write-up at\ndiffstudy.com\n, with every quote sourced. Papers are\narXiv 2211.17192\n(Leviathan et al.) and\narXiv 2302.01318\n(Chen et al.) if you'd rather go straight to them.",
  "metadata": {
    "title": "Speculative decoding won't change your model's distribution. It might still change your output. - DEV Community",
    "description": "There's a thread on the DeepSeek-R1 model page that's been sitting unresolved since March last year,... Tagged with computerscience, ai, machinelearning, llm.",
    "keywords": "computerscience, ai, machinelearning, llm, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/narotra05hp/speculative-decoding-wont-change-your-models-distribution-it-might-still-change-your-output-3de8",
    "og:title": "Speculative decoding won't change your model's distribution. It might still change your output.",
    "og:description": "There's a thread on the DeepSeek-R1 model page that's been sitting unresolved since March last year,...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fml7je91o7mzxdfsfy1no.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Speculative decoding won't change your model's distribution. It might still change your output.",
    "twitter:description": "There's a thread on the DeepSeek-R1 model page that's been sitting unresolved since March last year,...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fml7je91o7mzxdfsfy1no.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://diffstudy.com/speculative-decoding-vs-autoregressive-decoding/",
      "label": "diffstudy.com",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2211.17192",
      "label": "arXiv 2211.17192",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2302.01318",
      "label": "arXiv 2302.01318",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:01Z",
  "source_url": "https://dev.to/narotra05hp/speculative-decoding-wont-change-your-models-distribution-it-might-still-change-your-output-3de8",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2211.17192",
      "bytes": 574982,
      "reused": false,
      "local_path": "papers/repair/94fb44406a26971b792e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2302.01318",
      "bytes": 349916,
      "reused": false,
      "local_path": "papers/repair/37cfbe0b7967e7b005b9.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 5778,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/b102a0ad5a0c/pages/source.html",
  "local_html": "attachments/b102a0ad5a0c/pages/article.html",
  "body_base_path": "attachments/b102a0ad5a0c/pages/",
  "body_markdown_path": "attachments/b102a0ad5a0c/pages/article.md"
}