{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Seth Wheeler",
  "title_listed": "Predicting the Speed of a 276B Model Streamed From an SSD",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/megapixel99/predicting-the-speed-of-a-276b-model-streamed-from-an-ssd-50f8",
  "publish_date": "2026-08-29",
  "corpus_stage": "",
  "notes": "",
  "id": "0efa6fac1a90",
  "collected_at": "2026-10-02T13:51:59",
  "title": "Predicting the Speed of a 276B Model Streamed From an SSD - DEV Community",
  "body_markdown": "A mixture-of-experts model only activates a few of its experts per token, which means (unlike a dense model, where every forward pass touches every weight) you can leave the weights on disk and read in just the ones each token routes to. That turns \"does this model fit in RAM\" into \"how fast is your SSD,\" and it puts models far larger than your machine nominally supports within reach, at some token rate.\nI wanted to know what that rate actually is. The question that started it was concrete: can Inkling-Small (276B total parameters, 12B active) run on a 24 GB Mac mini?\nThe answer wanted to be arithmetic rather than a guess. This is what happened when the arithmetic met the machine.\nAct 1: a prediction is worthless unless it can be wrong\nI wrote a ~250-line cost model with no dependencies that downloads no weights. Every input is a\nconfig.json\nor\nmanifest.json\n; every output is a byte count derived from it. It models MLX affine group quantization the way it's actually stored on disk: per quantized projection, the weight bytes plus one bf16 scale and one bf16 bias per group of 64 values along the input dimension.\nThe obvious failure mode is a model tuned until it agrees with the one thing you checked it against. So it's gated twice, against artifacts I had nothing to do with.\nGate 1: reproduce a real container byte-for-byte.\nSwiftlet\nis a third-party Swift/Metal runtime that stores MoE weights in its own on-disk format. From a model's\nconfig.json\nalone, predict the layout it publishes. These are exact integers, so the gate demands equality, not closeness:\nquantity\ncomputed\npublished\nexpert stride\n1,769,472\n1,769,472\nexperts per layer\n256\n256\nper-layer blob bytes\n452,984,832\n452,984,832\nlayer count\n40\n40\nAll four exact. The full container total came within 0.137%, the residual being the tokenizer and chat template, which the model doesn't attempt.\nGate 2: predict a stranger's conversion.\nGate 1 could still pass by being fitted to the one container it was checked against. So: predict the on-disk size of a different model, converted by different people using a different tool.\nPredicted 151.8 GB against a published 153.5 GB, 1.1% off\n, against a 2% tolerance. The residual is unmodelled and deliberately not fudged: vision and audio encoders, RMS norms, 8-bit router gates, safetensors headers.\nCorrection, 2026-08-14.\nThis gate no longer passes, and the 1.1% was two errors cancelling. The model assumed one dense-MLP layer where the published model's own header states two, so it counted 41 mixture-of-experts layers instead of 40. Correcting that drops the prediction to 148.23 GB, which is\n3.4% off and outside the 2% tolerance\n. The extra layer had been inflating the prediction by 3.54 GB while the genuinely unmodelled components listed above deflated it by 5.27 GB, so the two nearly cancelled and the residual was always larger than 1.1% suggested. Gate 1 above is unaffected and still exact.\nThe follow-up post works through it\n.\nBoth gates run on every invocation, and the script refuses to print any token-rate estimate if either one fails. That mattered later.\nThe interesting structural result was that total parameter count is nearly irrelevant. What sets the per-token cost is\nexperts_per_token × layers × expert_size\n. And the useful finding was a cliff rather than a gradient: the fraction of each model's expert set that fits in 24 GB of RAM:\nmodel\ncacheable in 24 GB\nstreamed per token\nQwen3.6-35B-A3B\n93.7%\n0.566 GB\nInkling-Small (276B)\n10.0%\n3.397 GB\nCorrection, 2026-08-14.\nThat last figure read\n3.482 GB\nwhen this post went up, for the layer-count reason described above. The corrected 3.397 GB raises the predicted token rate quoted later in this post by a factor of 1.025, which is far too small to affect the 23× conclusion. The Qwen row is unaffected.\nThat 93.7% explains something I'd otherwise have taken at face value. Swiftlet's README reports its decode loop is \"dispatch bound, not IO bound,\" which sounds like a claim about kernel quality. On a 24 GB machine, after warmup, almost all of the 35B's experts are resident; it is barely streaming at all. The streaming machinery is what lets it\nstart\nin a couple of gigabytes, not what it lives in. Past the cliff, no amount of kernel work helps.\nNow I just needed a GB/s number to turn bytes per token into tokens per second.\nAct 2: the benchmark was measuring RAM\nAn earlier experiment of mine had priced disk reads at \"a nominal 100 µs\" and said so in its own limitations; the latency was an assumption, not an observation. So before trusting any rate, I measured the SSD.\nThe first version reported\n11–24 GB/s on a device that does about 7\n.\nThe cause is a specific and easy misreading of a macOS API.\nfcntl(fd, F_NOCACHE, 1)\nstops the kernel from caching\nfuture\nreads on that descriptor. It does not evict pages that are already resident. I was reading a file I had just written, so it was entirely in the unified buffer cache. The flag was set, it did exactly what it says, and the benchmark measured memory bandwidth.\nWhat makes this the interesting half is that no flag fixes it. The fix has to be structural: write a file\n2.5× larger than physical RAM\n(160 GiB on this machine) and sample only from its\nfirst quarter\n, whose pages the later 120 GiB of writes necessarily evicted.\nI also stopped trusting myself and added two gates that run every time:\nContent gate\n: every timed read must contain a magic header written at that offset. This catches short reads, misalignment, and filesystem tricks returning zeros at implausible speed. It immediately caught a page-alignment bug: 8 of 384 reads verified.\nCache gate\n, read the same offsets twice. With caching genuinely off, the two passes must agree. If pass 2 is much faster, the cache is serving reads and the run is void. It came out at 1.10×.\nThe real numbers, in the access pattern a router actually produces (top-k random experts inside one layer's blob, then the next layer) were\n3.54 GB/s issued serially and 7.82 GB/s with six reads in flight\n.\nThat 2.2× is the single biggest lever in the whole exercise. Issuing the top-k expert reads one at a time leaves half the device idle. For Inkling-Small it is the difference between roughly 1.1 and 2.5 tok/s: between unusable and marginal.\nA separate measurement at 4 KB found something worth designing against: the mean read is 154.1 µs, but\np99 is 757.6 µs, 5.8× the median\n. A mean-based estimate is structurally blind to that, and any paging design needs a deadline policy that answers from whatever is resident rather than waiting on the tail.\nThe generalizable version of this act:\nwhen a mechanism is supposed to disable an optimization, verify the optimization is actually off rather than trusting the switch.\nI had a flag that did what it promised, and a benchmark that was wrong by 3×.\nAct 3: running it, and missing by 23×\nPrediction, with the measured SSD:\n1.46–3.22 tok/s\n.\nGetting to a real number was mostly an exercise in discovering that published weights are not evidence that anything can load them. Four plausible runtimes:\npath\nstatus\nSwiftlet\nDifferent architecture family. A port, not a config change.\nMLX / mlx-lm\nmlx-lm\n0.29.1 ships 100 model modules and this architecture isn't one n/a despite a converted 4-bit build existing on HuggingFace.\nllama.cpp upstream\nRelease b10288's\nlibllama.dylib\ncontains\nzero\noccurrences of the architecture name. The GGUFs declare it.\nllama.cpp PR #25731\nThe only path. Still open, 9 commits, 63 files.\nThree of those four have published quantized weights and no working loader. The GGUF ladder exists because its author built it against their own unmerged branch.\nIt runs. On an M1 Max, CPU-only, streaming 127.4 GB from SSD via mmap:\nmeasured\npredicted\ndecode rate\n0.064 tok/s\n(15,678 ms/token)\n1.46–3.22 tok/s\npaged in from disk\n59.5 GB over 33 tokens\nn/a\n→ per generated token\n1.80–3.96 GB\n2.926 GB\n✓\nThe bytes side held.\n14,516,912 page-ins over 18 prompt plus 15 generated tokens brackets the predicted 2.926 GB/token: 3.96 GB if every page-in is charged to the generated tokens, 1.80 GB if spread across all 33. The batched prompt routes to far more than six experts per layer, so the true figure sits inside that bracket. Config arithmetic and kernel page-in counters are entirely independent methods, and they agree. That's the thing the experiment existed to get.\nThe rate side missed by 23×.\nAnd here is the part I actually care about.\nAct 1's write-up carried this in its limitations, written months before any of this ran:\ncompute is not modelled at all; every rate here is an IO ceiling, the real rate is the minimum of IO and compute, and this experiment cannot tell you which binds.\nThat is precisely what happened. The only configuration that completes a forward pass is CPU-only: every Metal configuration dies with an out-of-memory error, because ~125 GB of experts leak onto a GPU with a 51.5 GB working set, and the flag that's supposed to keep them off it doesn't work for this architecture. So 12B active parameters get multiplied on 8 CPU threads with no GPU, and\nthat\nsets 15.7 seconds per token. The SSD was never the constraint.\nThe IO ceiling was never wrong. It was just not the binding term in the only configuration available, and the run that works is therefore also the run that cannot test the ceiling. The 24 GB mini's predicted rate remains unmeasured, and I'd rather say that than quote 0.064 tok/s as though it answered the question.\nThe part that was worth the whole exercise\nThere's an operational footnote I'll keep short: a process streaming an oversized mmap'd model can become genuinely unkillable, sitting in uninterruptible-exiting state for 40 minutes after\nSIGKILL\n, because it cannot finish unwinding 127 GB of mapped memory while blocked in the page-fault path.\npkill\nreports success. Only\nps -o stat=\nshows otherwise. On unified memory this starves everything else on the machine, which is a good reason not to do this on a box something else depends on.\nBut the thing I'd take away is about the caveat.\nThe estimate was not the valuable output. Estimates get superseded. What made this worth doing is that the failure was\nattributable\n: the prediction missed for a reason that had been written down, in advance, as the thing the model could not account for. I didn't have to reconstruct an explanation after the fact, and I didn't get to choose a flattering one.\nThat is the difference between a model that taught me something and a number that happened to be wrong. A caveat written before the measurement is a prediction about your own blind spot; a caveat written after it is an excuse. They read identically in the final write-up, which is exactly why the order matters.\nI've now had this pattern three times in a row. A crawler whose rate limiter enforced a perfect delay while sending a hundred simultaneous requests. A browser whose hibernated tabs cost three orders of magnitude more than I'd modelled. And a cost model that was right about every byte and wrong about the clock. In all three the arithmetic was fine and the boundary of the arithmetic was where the truth was hiding, so the useful discipline isn't being more careful inside the model, it's being explicit about where the model stops.",
  "metadata": {
    "title": "Predicting the Speed of a 276B Model Streamed From an SSD - DEV Community",
    "description": "A from-config cost model for streaming a mixture-of-experts model off an SSD, validated byte-for-byte against a runtime I didn't write. Then the SSD benchmark turned out to be measuring RAM, and the real run missed the speed prediction by 23×, for the one reason the model had declared it couldn't account for. Tagged with llm, measurement, macos, ssd.",
    "keywords": "llm, measurement, macos, ssd, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/megapixel99/predicting-the-speed-of-a-276b-model-streamed-from-an-ssd-50f8",
    "og:title": "Predicting the Speed of a 276B Model Streamed From an SSD",
    "og:description": "A from-config cost model for streaming a mixture-of-experts model off an SSD, validated byte-for-byte against a runtime I didn't write. Then the SSD benchmark turned out to be measuring RAM, and the real run missed the speed prediction by 23×, for the one reason the model had declared it couldn't account for.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftc1wj2u98nobqcagtahs.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Predicting the Speed of a 276B Model Streamed From an SSD",
    "twitter:description": "A from-config cost model for streaming a mixture-of-experts model off an SSD, validated byte-for-byte against a runtime I didn't write. Then the SSD benchmark turned out to be measuring RAM, and the real run missed the speed prediction by 23×, for the one reason the model had declared it couldn't account for.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftc1wj2u98nobqcagtahs.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/leonickson1/Swiftlet",
      "label": "Swiftlet",
      "type": "a"
    },
    {
      "href": "https://sethwheeler.dev/blog/cancelling-errors/",
      "label": "The follow-up post works through it",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:41Z",
  "source_url": "https://dev.to/megapixel99/predicting-the-speed-of-a-276b-model-streamed-from-an-ssd-50f8",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 11244,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/0efa6fac1a90/pages/source.html",
  "local_html": "attachments/0efa6fac1a90/pages/article.html",
  "body_base_path": "attachments/0efa6fac1a90/pages/",
  "body_markdown_path": "attachments/0efa6fac1a90/pages/article.md"
}