{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "A Better Model Improved the Numbers. It Didn't Fix the Product.",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/better-models-showed-us-what-to-build-next-1oj6",
  "publish_date": "2026-09-07",
  "corpus_stage": "",
  "notes": "",
  "id": "c74d150ca88e",
  "collected_at": "2026-10-02T14:06:19",
  "title": "A Better Model Improved the Numbers. It Didn't Fix the Product. - DEV Community",
  "body_markdown": "Update — v0.1.0 released.\nCauterRule is now live on\nGitHub\nand\nPyPI\n. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.\npip install cauterule\ngives you 25+ CLI commands, an MCP server, 7 export formats, and a bundled git rule pack. The\nfield test report\nis the source for every number in this article: 4 models, 13 corpus types, 394 trajectories, 844+ deterministic tests passing. The strongest cloud model tested was\nmeta-llama/llama-3.1-8b-instruct\nwith 123 pass / 168 inconclusive / 101 fail — better than every other model, but still not enough to solve the safety problem on its own.\nRelease notes\n·\nChangelog\nThere is a comforting fantasy in AI engineering. When a system struggles, you tell yourself the problem is the model. Spend a little more, use a stronger one, and the product gets better.\nSometimes that is true. One of the most useful outcomes from the CauterRule field test was learning exactly where that story stops being true.\nWhy we ran cloud models\nAfter fixing the parser, prompt, result-reset, and corpus timestamp issues, we finally had a benchmark we could trust. That was the moment cloud comparison became worth doing. We ran two working cloud models through the full corpus:\nopenai/gpt-4o-mini\nand\nmeta-llama/llama-3.1-8b-instruct\n.\nThe point was not just to get better numbers. It was to answer a sharper question:\nAre the remaining problems mostly local-model weakness, or are they product problems?\nThe answer turned out to be both, but mostly the second.\nAll-model comparison: local vs cloud\nThe broadest comparison is the full-corpus view.\nModel\nType\nTotal rows\nTotal candidates\nPass\nPass %\nInconclusive\nFail\nomlx-openai-Llama-3.2-3B-Instruct-4bit\nlocal\n393\n379\n72\n19.0%\n189\n118\nomlx-openai-Qwen3-4B-Instruct-2507-4bit\nlocal\n373\n373\n93\n24.9%\n209\n71\nopenai/gpt-4o-mini\ncloud\n394\n394\n77\n19.5%\n248\n69\nmeta-llama/llama-3.1-8b-instruct\ncloud\n394\n392\n123\n31.4%\n168\n101\nThree things jump out. First, the cloud models removed most of the remaining availability and formatting ambiguity — both produced near-perfect parse reliability. Second,\nmeta-llama/llama-3.1-8b-instruct\nproduced the strongest overall pass count of any model we tested: 123 passes from 392 candidates. Third,\ngpt-4o-mini\nwas still operationally valuable because it was stable, cheap, and fully runnable across all 394 trajectories.\nBut there is something more subtle in this table that matters more than the ranking. Even the strongest tested model still produced 168 inconclusives and 101 hard fails. That is 269 out of 392 candidates —\n69% of the output was not clearly usable\n. Better models improved the system. They did not settle the product.\nCurated comparison: where model quality actually mattered most\nThe curated corpora are the cleanest lens for this comparison.\nCurated corpus\nLocal Llama 3.2B\nLocal Qwen 4B\nCloud GPT-4o-mini\nCloud Llama 3.1 8B\ngolden\n8P / 2F\n3P / 4I / 3F\n6P / 4I / 0F\n7P / 0I / 3F\nfailures/positive\n15P / 3I / 12F\n15P / 9I / 6F\n14P / 13I / 3F\n22P / 5I / 3F\nfailures/negative\n1P / 2I / 5F\n1P / 4I / 5F\n0P / 4I / 6F\n3P / 1I / 6F\nsuccesses\n1P / 16I / 2F\n2P / 8I / 10F\n0P / 6I / 14F\n3P / 6I / 11F\nnearmiss\n4P / 4I / 6F\n5P / 5I / 4F\n2P / 4I / 8F\n6P / 1I / 7F\nnoisy\n1P / 3I / 1F\n4P / 1I / 0F\n1P / 3I / 1F\n5P / 0I / 0F\ncorrections\n2P / 2I / 1F\n2P / 1I / 2F\n3P / 2I / 0F\n3P / 1I / 1F\nIf you want the shortest honest reading of that table: cloud models clearly help on quality.\nmeta-llama/llama-3.1-8b-instruct\nwas the strongest model tested. And the positive-case corpora improved much faster than the safety corpora.\nThat last point matters more than the ranking itself.\nWinner-by-corpus view\nThis view is a better way to summarize what actually improved.\nCorpus\nBest model\nWhy it matters\ngolden\nLocal Llama 3.2B by raw pass count, Cloud Llama 3.1 8B by strongest cloud showing\nregression anchor / specificity\nfailures/positive\nCloud Llama 3.1 8B\nbest practical extraction benchmark\nsuccesses\nnone were convincingly strong\nsafety / over-triggering check\nfailures/negative\nnone were convincingly strong (all 5-6 fails)\nrejection quality\nnearmiss\nCloud Llama 3.1 8B, but still mixed\ntrigger precision\nnoisy\nCloud Llama 3.1 8B\nstrongest clutter-handling\ncorrections\nCloud GPT-4o-mini and Cloud Llama 3.1 8B both strong\ncorrection-to-rule synthesis\nThe cloud models won where learning-from-failure was easiest. They did not decisively win where restraint mattered most.\nThe raw corpus results confirm the pattern\nThe raw corpora add volume and show the same pattern at scale.\nRaw corpus\nLocal Llama 3.2B\nLocal Qwen 4B\nCloud GPT-4o-mini\nCloud Llama 3.1 8B\nraw/opencode\n13P / 2I / 9F\n13P / 7I / 5F\n12P / 10I / 3F\n18P / 3I / 4F\nraw/synthetic\n18P / 101I / 26F\n36P / 87I / 22F\n28P / 91I / 26F\n38P / 81I / 25F\nraw/ci\n8P / 49I / 52F\n9P / 90I / 11F\n7P / 99I / 4F\n11P / 60I / 39F\nraw/sibling-repos\n0P / 9I / 1F\n0P / 9I / 1F\n0P / 7I / 3F\n0P / 10I / 0F\nraw/corrections\n2P / 2I / 1F\n3P / 1I / 1F\n2P / 3I / 0F\n3P / 0I / 1F\nraw/cross-session\n2P / 1I / 2F\n1P / 2I / 2F\n2P / 2I / 1F\n4P / 0I / 1F\nCloud Llama 3.1 8B led on\nraw/opencode\n(18P),\nraw/synthetic\n(38P),\nraw/corrections\n(3P), and\nraw/cross-session\n(4P). But on\nraw/ci\n, it still produced 60 inconclusives and 39 hard fails from 110 trajectories. That is a lot of noise from the strongest model in the batch.\nThe report makes an observation about this that is worth pulling out: raw breadth without replay clarity is not yet evidence. The product can\nprocess\nbroad raw corpora, but if it mostly yields inconclusive judgments, that is not the same as producing trustworthy results. Broad processing coverage is good. But broad coverage that produces mostly inconclusives is not yet the strongest evidence of product trustworthiness.\nWhat the cloud models clearly improved\nThe cloud runs gave us much cleaner evidence than the early local-only picture:\nnear-perfect parse reliability\ncomplete candidate generation across the corpus\nstronger results on\ngolden\nand\nfailures/positive\nstronger performance on\ncorrections\nand broad raw corpora\nOne model stood out:\nmeta-llama/llama-3.1-8b-instruct\nwas the strongest cost-effective model in the entire batch. On\nfailures/positive\n, it produced 22 pass / 5 inconclusive / 3 fail — the best single result on the most practical extraction benchmark. On\nnoisy\n, it produced 5 pass / 0 inconclusive / 0 fail — a clean sweep. On\nraw/opencode\n, it produced 18 pass — more than any other model.\nThe report's verdict: best current cost/performance benchmark model. Not because it was the most expensive or the largest, but because it consistently outperformed\ngpt-4o-mini\non the corpora that matter most for practical rule extraction.\nWhat better models did not fix\nThis is the more important half of the story.\nEven with stronger cloud models, the system still struggled on the corpora that matter most for trust:\nsuccesses\n,\nfailures/negative\n, and\nnearmiss\n.\nThe simplest local-vs-cloud safety view:\nSafety-sensitive corpus\nBest local outcome\nBest cloud outcome\nWhat it suggests\nsuccesses\nstill weak\nstill weak\nreplay and restraint remain product issues\nfailures/negative\nnone were convincingly strong\nnone were convincingly strong\nstronger generation does not solve rejection quality\nnearmiss\nmixed\nmixed\ntrigger specificity remains a hard problem\nOn\nsuccesses\n, the strongest cloud model produced 3 pass / 6 inconclusive /\n11 fail\n. Those 3 \"passes\" are not a win — they are false positives. A \"pass\" on the\nsuccesses\ncorpus means the system extracted a rule from a trajectory that should not have produced one. Every pass on\nsuccesses\nis a safety violation.\nOn\nfailures/negative\n, the strongest cloud model produced 3 pass / 1 inconclusive / 6 fail. Every model in the field test produced 5-6 hard fails on this corpus, and no model produced more than 3 passes. The rejection gate is weak regardless of which model drives extraction.\nOn\nnearmiss\n, Cloud Llama 3.1 8B produced 6 pass / 1 inconclusive / 7 fail. Better than the locals, but still more failures than passes. The trigger is firing on cases that look similar but should not trigger.\nIf cloud models had cleaned those up, I would be writing a very different article. They did not.\nThe gap is not model quality — it is judgment quality\nThis is the insight that changed how I think about the roadmap.\nBefore the cloud runs, it was possible to believe the remaining weakness was mostly a model problem. Small local models, limited instruction-following, noisy output — maybe a bigger model would fix it.\nAfter the cloud runs, that explanation is dead. A stronger model with near-perfect parse reliability, running on a now-trustworthy benchmark, still produced 11 fails on\nsuccesses\nand could not reliably reject bad candidates on\nfailures/negative\n. The problem is not that the model cannot generate rules. The problem is that the product's judgment — when to extract, when to reject, when to stay silent — is not strong enough.\nThe report identifies six specific gaps between the current state and a stronger release claim:\nSafety gap\n— the system performs too weakly on\nsuccesses\n,\nfailures/negative\n, and\nnearmiss\n.\nReplay-trust gap\n— the matcher and replay engine produce too many inconclusives, especially on raw corpora.\nPromotion-confidence gap\n— many plausible candidates, but not enough evidence they are consistently safe to promote.\nHuman-judgment gap\n— the report leans on replay heuristics more than human-reviewed rule quality.\nRelease-criteria gap\n— no explicit, enforced pass/fail thresholds tied to safety-sensitive corpora.\nOperational-comparison gap\n— we know which models are promising, but need a crisper policy for which model class qualifies as authoritative evidence.\nThe first three are the real release blockers. The others matter, but safety, replay trust, and promotion confidence are what define whether the product can honestly claim v0.1.0 field-test success in the strongest sense.\nWhat I learned from this\nBetter models improve the ceiling, not the floor.\nThe cloud models lifted the best results — more passes on\nfailures/positive\n, cleaner\nnoisy\nhandling, stronger raw corpus extraction. But they did not lift the worst results.\nsuccesses\nand\nfailures/negative\nstayed weak. That tells you the floor is a product problem, not a model problem.\nA \"pass\" on a safety corpus is not always a win.\nOn\nsuccesses\n, a pass means the system extracted a rule from a trajectory that should not have produced one. Counting passes without reading the corpus label gives you a false sense of progress. You have to know what each corpus is testing before you interpret the number.\nInconclusives are the silent killer.\ngpt-4o-mini\nproduced 248 inconclusives out of 394 candidates — 63% of its output was uncommitted. Those inconclusives did not show up as failures, so they did not trigger alarm. But they also did not produce usable rules. A model that is 63% inconclusive is not producing 77 passes. It is producing 77 passes and 248 maybes. Maybes do not build trust. And the report is clear that some of those maybes are a replay-engine limitation, not a model limitation — the matcher is the named #1 bottleneck. But on the safety corpora, inconclusives are a different signal: they mean the model extracted something from a trajectory where it should have stayed silent, and the matcher could not cleanly reject it. Those are model problems hiding behind engine weakness. You have to split the inconclusive bucket by corpus before you know which ones are the matcher's fault and which ones are the model's.\nThe field test succeeded even though the product did not fully pass.\nThis sounds contradictory but it is not. The field test succeeded because it forced the system through realistic comparisons, exposed tooling defects, validated that fixes improved signal, and produced a coherent map of what remains. The product improved materially. The product is not yet fully field-test complete by a strict release gate. Both of those things are true. That is a good outcome for an honest engineering report.\nOpen questions\nWould a much larger model (70B, 405B) close the safety gap, or is the\nsuccesses\nproblem fundamentally about replay logic, not extraction quality? The 8B results suggest the latter, but we have not tested it.\nThe\nfailures/negative\ncorpus is small (8 trajectories per model). Would a larger negative corpus surface more rejection failures, or is 8 enough to see the pattern? The current results are already weak enough to be concerning at this scale.\ngpt-4o-mini\nproduced zero fails on\ngolden\n(6P / 4I / 0F) but 14 fails on\nsuccesses\n(0P / 6I / 14F). That is a striking contrast. Is the model genuinely good at extraction and genuinely bad at restraint, or is the\nsuccesses\ncorpus testing something the model was never designed to handle?\nThe report recommends safety-first quality gates: promote only when\nsuccesses\n,\nfailures/negative\n, and\nnearmiss\nsupport the claim. What happens to the promotion rate if we enforce that? If it drops to near zero, that tells us the current candidate pool is not safe enough. If it stays nonzero, that gives us a credible set of rules to build on.\nWhy this is actually good news\nIf the cloud models had solved everything, the conclusion would have been easy: just use a better model. That would be convenient, but it would teach us less.\nWhat happened instead was better. The cloud runs improved the benchmark enough to expose the real work that remains. They showed that stronger models help a lot, but they also showed that the core challenge now lives in replay trust, safety, and promotion confidence.\nThat is exactly what a serious field test should surface.\nThe cloud models made the product look better. They also made its remaining weaknesses harder to deny. That is why we ran them.\nCauterRule v0.1.0 is released.\nThe full all-model comparison — local vs cloud, curated and raw corpora, winner-by-corpus view, and model recommendations — is in the\nfield test report\n. The\nrepo\nis public. Install with\npip install cauterule\n.\nChangelog\n·\nRelease notes",
  "metadata": {
    "title": "A Better Model Improved the Numbers. It Didn't Fix the Product. - DEV Community",
    "description": "Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent... Tagged with ai, llm, testing, programming.",
    "keywords": "ai, llm, testing, programming, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/better-models-showed-us-what-to-build-next-1oj6",
    "og:title": "A Better Model Improved the Numbers. It Didn't Fix the Product.",
    "og:description": "Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmuc11mppg6mq77rxxb1q.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "A Better Model Improved the Numbers. It Didn't Fix the Product.",
    "twitter:description": "Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmuc11mppg6mq77rxxb1q.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/deghosal-2026/CauterRule",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://pypi.org/project/cauterule/",
      "label": "PyPI",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.1.0/FIELD_TEST_REPORT.md",
      "label": "field test report",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/releases/tag/v0.1.0",
      "label": "Release notes",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md",
      "label": "Changelog",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:19Z",
  "source_url": "https://dev.to/debashish_ghosal/better-models-showed-us-what-to-build-next-1oj6",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 14039,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/c74d150ca88e/pages/source.html",
  "local_html": "attachments/c74d150ca88e/pages/article.html",
  "body_base_path": "attachments/c74d150ca88e/pages/",
  "body_markdown_path": "attachments/c74d150ca88e/pages/article.md"
}