{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "We Could Have Shipped on Local Models Alone",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/small-local-models-earned-their-place-1bl5",
  "publish_date": "2026-09-06",
  "corpus_stage": "",
  "notes": "",
  "id": "9f9ed6e8c2fe",
  "collected_at": "2026-10-02T14:06:21",
  "title": "We Could Have Shipped on Local Models Alone - DEV Community",
  "body_markdown": "Update — v0.1.0 released.\nCauterRule is now live on\nGitHub\nand\nPyPI\n. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.\npip install cauterule\ngives you the full CLI, MCP server, 7 export formats, and a bundled git rule pack. The\nfield test report\nevaluated 4 models (2 local OMLX, 2 cloud OpenRouter) across 394 trajectories. The local-model totals below — Llama 3.2B at 379 candidates and Qwen 4B at 373 — come directly from that report.\nRelease notes\n·\nChangelog\nCauterRule\nis an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.\nOne of the easiest mistakes to make in AI engineering is asking the wrong question. We started our field test with this one:\nAre small local models good enough?\nBy the end of the work, the more useful question looked different:\nWhat are small local models good enough for?\nAnd after the cloud runs came back, a third question surfaced that I did not expect to be able to answer affirmatively:\nIf I only had local models, could I still ship this product with the same confidence?\nThe answer turned out to be yes. Not because local models are as strong as the best cloud model — they are not. But because the thing that blocks shipping confidence is not the model. It is the replay engine and the safety corpora, and cloud models do not fix those. More on that below.\nTwo models did not even make it to the benchmark\nBefore the comparison could start, two candidate models were cut.\nQwen3.5-4B-4bit\nwas too slow for practical batch work — a model that takes minutes per trajectory is not one you can run repeatedly across 394 trajectories.\ngoogle/gemini-2.0-flash-001\nwas unavailable on the tested OpenRouter tier. The endpoint simply was not there.\nThat is an operational detail that benchmark papers often skip. Two out of six candidates were eliminated before a single extraction ran. Model selection needs an operational screen — speed, availability, cost — before you invest in a full corpus sweep. Skip that screen and you waste real testing time on models that were never going to work in your workflow.\nThe field test report puts this bluntly: not every model failure is a product signal. Some are simply deployment or availability problems. The report needs to separate those cleanly so they do not pollute the product conclusions.\nThe post-fix local results were better than the early benchmark implied\nMost of what the early benchmark called \"local model failures\" were not model failures at all. They were benchmark failures. The field test report puts this bluntly: before the fixes, both local models showed\nsevere parse failure rates\non curated corpora. After the fixes, parse failures dropped to\nnear-zero\non most curated corpora. The models did not change. The benchmark did.\nWhat was broken\nBefore\nAfter\nResult-file isolation\nSame-day reruns appended duplicate rows, contaminating counts\nReruns reset result files at start\nJSON extraction\nResponses with trailing commentary or a second JSON object were rejected\nFirst balanced JSON object is now extracted\nwhen.context\nvalidation\nBlank context entries caused hard parse failure\nBlank entries filtered before validation\nPrompt clarity\nMinimal example encouraged inconsistent formatting\nConcrete filled example + explicit \"JSON only\" instruction\nCurated parse reliability\nSevere parse failure rates on both local models\nNear-zero parse failures across most curated corpora\nRaw corpus completeness\nraw/synthetic\nand\nraw/sibling-repos\nblocked by missing timestamps\nTimestamp fix applied; both corpora run end to end\nThat is six separate harness defects — none of them in the models — that were collectively making the local models look worse than they were. When the early benchmark said \"small local models are borderline unusable,\" the honest reading was \"small local models are borderline unusable\nthrough this broken benchmark\n.\" Fix the benchmark and the same models produce 379 and 373 candidates across 13 corpus types.\nThis is the part of the story that is easy to miss if you only look at the final numbers. The early narrative was not \"local models have some weaknesses.\" It was \"local models are mostly failing.\" That narrative was wrong, and the reason it was wrong is that the evaluation harness was failing the models, not the other way around.\nAfter the fixes, the OMLX models became much more meaningful engineering tools. The cleanest curated comparison:\nCurated corpus\nLlama 3.2B\nQwen 4B\ngolden\n8 pass / 2 fail\n3 pass / 4 inconclusive / 3 fail\nfailures/positive\n15 pass / 3 inconclusive / 12 fail\n15 pass / 9 inconclusive / 6 fail\nfailures/negative\n1 pass / 2 inconclusive / 5 fail\n1 pass / 4 inconclusive / 5 fail\nsuccesses\n1 pass / 16 inconclusive / 2 fail\n2 pass / 8 inconclusive / 10 fail\nnearmiss\n4 pass / 4 inconclusive / 6 fail\n5 pass / 5 inconclusive / 4 fail\nnoisy\n1 pass / 3 inconclusive / 1 fail\n4 pass / 1 inconclusive / 0 fail\ncorrections\n2 pass / 2 inconclusive / 1 fail\n2 pass / 1 inconclusive / 2 fail\nNot a story of \"small local models solved the problem.\" But also not a story of \"small local models were worthless.\" The full benchmark totals reinforce that.\nModel\nTotal rows\nTotal candidates\nPass\nPass %\nInconclusive\nFail\nLocal Llama 3.2B\n393\n379\n72\n19.0%\n189\n118\nLocal Qwen 4B\n373\n373\n93\n24.9%\n209\n71\n379 candidates from a 3B model running locally on a laptop. 373 from a 4B model. Those are large enough run volumes to support real engineering decisions, not just gut feelings.\nBut the raw pass rate only tells part of the story. To see how local models actually stack up, you have to put them next to the cloud baselines from the same field test:\nModel\nType\nCandidates\nPass\nPass %\nLocal Llama 3.2B\nlocal OMLX\n379\n72\n19.0%\nLocal Qwen 4B\nlocal OMLX\n373\n93\n24.9%\nCloud GPT-4o-mini\ncloud OpenRouter\n394\n77\n19.5%\nCloud Llama 3.1 8B\ncloud OpenRouter\n392\n123\n31.4%\nRead that table carefully. Local Qwen 4B — a 4B model running on a laptop — outperformed\nopenai/gpt-4o-mini\non total passes (93 vs 77) and on raw pass rate (24.9% vs 19.5%). Local Llama 3.2B beat GPT-4o-mini on the\ngolden\ncorpus (8 pass vs 6 pass). This is not \"small local models earned a place as a regression tool.\" This is \"a 4B local model beat a paid cloud baseline head-to-head on total pass count.\"\nThe cloud Llama 3.1 8B is still the strongest model in the field test, and the safety corpora remain weak across the board. But the premise that local models are clearly inferior to cheap cloud baselines is not supported by this data. On some corpora, the local models are better.\nAnd here is the point that matters most for anyone deciding whether to ship: the cloud runs did not move the shipping decision. The field test report says it plainly —\n\"better models improve extraction quality, but they do not erase the product's safety and replay problems.\"\nThe strongest cloud model still fails on\nsuccesses\n,\nfailures/negative\n, and\nnearmiss\n. The replay engine is still the named #1 bottleneck. The safety problem lives in the product, not in the model. So if your shipping gate is \"do we have enough confidence in the safety and replay layer,\" cloud models do not get you over that line any more than local models do. They get you a cleaner extraction ceiling, not a safer product.\nThat is why the honest answer to \"could I ship with local models only?\" is yes. You would ship with the same unresolved safety and replay gaps either way. The cloud model does not close them.\nWhat the pass counts are actually measuring\nThere is one more thing the headline numbers do not show, and it matters for how you read every table in this article. The\npass / inconclusive / fail\nverdict comes from the replay engine, not from a human judge. And the field test report names that replay engine as the single biggest bottleneck in the whole system:\nReplay and matcher calibration is now the highest-value engineering target.\nThe matcher uses substring and token-overlap heuristics. When it cannot decide whether a rule fires against a reference trajectory, it returns\ninconclusive\n— not\nfail\n. That means a large slice of the inconclusive bucket is not a model failure. It is a replay-engine limitation. A specific, reasonable rule can land in\ninconclusive\npurely because the matcher is too weak to verify it.\nSo the pass counts in the tables above are closer to a\nfloor\non model usefulness than a ceiling. The true model-only pass rate sits somewhere between the reported pass count and pass-plus-inconclusive. For Qwen 4B, that range is 24.9%–81.0% of candidates. For Llama 3.2B, it is 19.0%–68.9%.\nThat is a wide band, and it is not honest to claim all inconclusives as hidden passes. Some of them are real model weaknesses — over-broad triggers like\n\"when a command fails\"\ninstead of\n\"when git push fails with non-fast-forward\"\n, and over-triggering on the\nsuccesses\ncorpus where the model should have stayed silent. Those are model problems, not matcher problems.\nBut it is also not honest to present the raw pass count as the model's ceiling. The report is explicit: the replay engine is the named bottleneck, and better models did not erase the inconclusive problem. So when you read that Local Qwen 4B passed 93 of 373 candidates, the honest interpretation is \"at least 93, and some unknown fraction of the 209 inconclusives are also correct extractions that the matcher could not verify.\"\nThe raw corpus split: Llama wins breadth, Qwen wins depth\nThe curated corpora showed Llama as the better default. The raw corpora told a more nuanced story.\nRaw corpus\nLocal Llama 3.2B\nLocal Qwen 4B\nraw/opencode\n13P / 2I / 9F\n13P / 7I / 5F\nraw/synthetic\n18P / 101I / 26F\n36P / 87I / 22F\nraw/ci\n8P / 49I / 52F\n9P / 90I / 11F\nraw/sibling-repos\n0P / 9I / 1F\n0P / 9I / 1F\nraw/corrections\n2P / 2I / 1F\n3P / 1I / 1F\nraw/cross-session\n2P / 1I / 2F\n1P / 2I / 2F\nLook at\nraw/synthetic\n: Qwen produced\n36 pass\nvs Llama's\n18 pass\n— double the extracted rules. But Llama had 26 hard fails vs Qwen's 22. And both produced a large number of inconclusives (101 vs 87).\nThen\nraw/ci\n: Llama produced 8 pass / 49 inconclusive /\n52 fail\n. Qwen produced 9 pass / 90 inconclusive /\n11 fail\n. Qwen had fewer hard failures but far more inconclusives. That is a different quality profile — Qwen is more cautious, Llama is more decisive but also more wrong.\nThis is why raw pass count alone is not enough to pick a default local model. The\npattern\nof pass / inconclusive / fail matters. High pass + high fail means decisive but noisy — you get more signal but more garbage. Moderate pass + high inconclusive + low fail means cautious — less garbage but less signal. For a local regression tool, you want the decisive model — false positives are cheaper because you can review them. For a trust benchmark, you want the cautious one — false positives are expensive because they can be promoted.\nWhat local models were actually good for\nAfter the fixes, the local models were useful for:\nbroad internal corpus sweeps\nlow-cost repeated reruns\nprompt and parser regression checks\nlocal smoke tests for the extraction loop\ncheap model-to-model comparisons\nThat is already a meaningful role. And once you look at the local-vs-local comparison in a practical way, the split becomes clearer:\nQuestion\nLocal Llama 3.2B\nLocal Qwen 4B\nBetter practical local default?\nYes\nNo\nBetter secondary comparison model?\nAcceptable\nYes\nBetter on raw synthetic breadth?\nWeaker\nStronger\nBetter operational feel for reruns?\nYes\nSlower / heavier\nThis is why I do not think raw pass count alone is enough to pick the default local model. Engineering usefulness includes speed, rerun friction, and how easy a model is to keep in the loop.\nWhere local models were still clearly limited\nThe same results also make the limits obvious. The weakest areas remained the same ones that matter most for trust:\nsuccesses\n,\nfailures/negative\n, and\nnearmiss\n.\nThat means the local models are still not strong enough to be the only evidence behind public quality claims or autonomous promotion decisions. But neither are the cloud models — the report's central finding is that even\nmeta-llama/llama-3.1-8b-instruct\ndoes not solve the safety corpora. So the honest caveat is not \"local models are only good for regression.\" It is \"no model in this field test is sufficient for trust on its own, and the safety problem lives in the product, not in the model.\" Local models are competitive with cheap cloud baselines on extraction quality; what they all still lack together is safe rule selection and replay trust.\nAn observation that surprised me\nI expected the larger local model (Qwen 4B) to clearly outperform the smaller one (Llama 3.2B). It did not. On the curated corpora, Llama 3.2B actually produced more\ngolden\npasses (8 vs 3) and was more\ndecisive\non\nfailures/positive\n— 15 pass / 3 inconclusive / 12 fail, versus Qwen's 15 pass / 9 inconclusive / 6 fail. By raw pass-to-fail ratio, Qwen actually wins that corpus (15:6 ≈ 2.5 vs Llama's 15:12 ≈ 1.25). But Llama produced fewer inconclusives, which means it committed to a verdict more often. Qwen caught up on breadth — especially on\nraw/synthetic\nwhere it doubled Llama's pass count — but it did not translate that into better curated precision.\nThat suggests something I did not expect: for small local models, raw capability and extraction precision may not correlate the way you'd assume. A 3B model with good instruction-following can outperform a 4B model on specific tasks, even when the 4B model processes more data. The bottleneck at this scale is not parameters. It is instruction adherence and format stability.\nWhat I learned from this\nFirst impressions from a broken benchmark are worse than no benchmark at all.\nThe early data made local models look borderline unusable. After the fixes, they produced hundreds of candidates across 13 corpus types. The models did not change. The benchmark did. If we had trusted the early data, we would have written off local models entirely and spent money on cloud APIs for every regression run.\nOperational screening belongs before benchmarking, not after.\nWe lost time on Qwen3.5-4B (too slow) and Gemini Flash (unavailable). Both failures were predictable with a 5-minute operational check. A full corpus sweep across 394 trajectories is expensive. Do the cheap screening first.\nPass count is not the same as decision quality.\nQwen 4B had more total passes (93 vs 72) but also more inconclusives (209 vs 189). Llama 3.2B had more hard fails (118 vs 71). Which model is \"better\" depends on what you are using it for — a regression tool wants decisiveness, a trust benchmark wants caution.\nRole separation beats model replacement.\nThe temptation when a model is weak is to replace it. The better move is to assign it a role it can handle. But note what the data actually showed: a 4B local model beat a paid cloud baseline on total pass count. So \"local for iteration, cloud for quality\" is true, but it is not true because local models are weak. It is true because cloud models give you a cleaner quality ceiling and wider model choice, not because local models cannot compete on extraction. The cheapest useful cloud model (\nmeta-llama/llama-3.1-8b-instruct\n) still beat every local model tested — but\ngpt-4o-mini\ndid not.\nOpen questions\nWould a 7B or 8B local model close the gap on\nsuccesses\nand\nnearmiss\n, or does the safety problem live in the product regardless of model size? The cloud results in the next article suggest the latter.\nIs the Llama 3.2B advantage on\ngolden\nstable across reruns, or is it within noise? We ran each model once per corpus. More reruns would tell us whether 8P/2F is reliable or lucky.\nThe\nraw/synthetic\nsplit (Qwen 36P vs Llama 18P) is large enough to matter. Is Qwen genuinely better at breadth, or is it over-triggering on raw data? The inconclusive count (87 vs 101) suggests Qwen is extracting more but also hedging more.\nThe broader lesson\nLocal models were not good enough to end the evaluation conversation. But they were much better than their first benchmark impression — good enough, in fact, to beat a paid cloud baseline on total pass count, and good enough to ship the product on. The lesson is not that local models are a cheap fallback. It is that the gap between \"cheap local\" and \"cheap cloud\" is smaller than the benchmark infrastructure made it look, and the thing that blocks shipping confidence — the replay engine and the safety corpora — is a product problem that cloud models do not fix. If I only had local models, I could have shipped CauterRule with the same degree of confidence the cloud models gave. Which is to say: the same confidence, and the same unresolved gaps.\nCauterRule v0.1.0 is released.\nThe full model-by-model breakdown — including the curated corpus comparison table, raw corpus results, and model ranking — is in the\nfield test report\n. The\nrepo\nis public. Install with\npip install cauterule\n.\nChangelog\n·\nRelease notes",
  "metadata": {
    "title": "We Could Have Shipped on Local Models Alone - DEV Community",
    "description": "Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent... Tagged with ai, llm, healthydebate, testing.",
    "keywords": "ai, llm, healthydebate, testing, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/small-local-models-earned-their-place-1bl5",
    "og:title": "We Could Have Shipped on Local Models Alone",
    "og:description": "Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fll03og6j0lyln4zvnrc7.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "We Could Have Shipped on Local Models Alone",
    "twitter:description": "Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fll03og6j0lyln4zvnrc7.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/deghosal-2026/CauterRule",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://pypi.org/project/cauterule/",
      "label": "PyPI",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.1.0/FIELD_TEST_REPORT.md",
      "label": "field test report",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/releases/tag/v0.1.0",
      "label": "Release notes",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md",
      "label": "Changelog",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:20Z",
  "source_url": "https://dev.to/debashish_ghosal/small-local-models-earned-their-place-1bl5",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 17031,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/9f9ed6e8c2fe/pages/source.html",
  "local_html": "attachments/9f9ed6e8c2fe/pages/article.html",
  "body_base_path": "attachments/9f9ed6e8c2fe/pages/",
  "body_markdown_path": "attachments/9f9ed6e8c2fe/pages/article.md"
}