{ "source_file": "0930", "source_platform": "Dev.to", "author": "Debashish Ghosal", "title_listed": "When Your Benchmark Finally Tells the Truth", "keywords": "", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://dev.to/debashish_ghosal/when-your-benchmark-finally-tells-the-truth-534h", "publish_date": "2026-09-06", "corpus_stage": "", "notes": "", "id": "9be2da967e11", "collected_at": "2026-10-02T14:06:22", "title": "When Your Benchmark Finally Tells the Truth - DEV Community", "body_markdown": "Update — v0.1.0 released.\nCauterRule is now live on\nGitHub\nand\nPyPI\n. It turns repeated agent failures into permanent standing rules — extract a candidate rule from a failure trajectory, replay-test it against history, promote only what survives.\npip install cauterule\ngets you the full CLI, MCP server, export to 7 formats, and a bundled git rule pack. The\nfield test report\nevaluates 4 models across 394 trajectories and is the source for every number in this article.\nRelease notes\n·\nChangelog\nCauterRule\nis an open-source sidecar that turns repeated agent failures into standing rules. It extracts candidate lessons from trajectories, replay-tests them against history, and is designed to promote only rules that look reusable and safe.\nWhen we started field testing CauterRule, the early numbers looked bad enough that the story almost wrote itself. Local OMLX models looked weak. Parse failures were common. Some corpus slices would not run fully. It was tempting to conclude that the model layer was the main problem.\nThat was not the right conclusion.\nThe better conclusion, backed by the rerun data, was that the benchmark harness itself was manufacturing a meaningful part of the failure.\nThe before/after numbers were too large to ignore\nThe clearest evidence came from rerunning the same benchmark surfaces after parser, prompt, result-reset, and timestamp fixes.\nSignal\nBefore fixes\nAfter fixes\nLocal Llama\ngolden\nparsed candidates\n3 / 11 rows\n10 / 10 rows\nLocal Qwen\ngolden\nparsed candidates\n3 / 10 rows\n10 / 10 rows\nRaw synthetic processed rows\n124 / 145\n145 / 145\nRaw sibling-repo processed rows\n0 / 10\n10 / 10\nThose are not cosmetic improvements. A benchmark that can only parse 3 of 10 or 3 of 11\ngolden\nrows is measuring something very different from a benchmark that cleanly parses 10 of 10.\nTo put that in perspective: the early benchmark was throwing away\n73% of its own signal\non the\ngolden\ncorpus. It was not measuring model quality. It was measuring parser fragility and then labeling the result as \"model quality.\" If we had shipped a product roadmap based on that data, we would have optimized the wrong layer for months.\nAnd it was not just\ngolden\n. The raw synthetic corpus went from 124/145 to 145/145 processed rows. The raw sibling-repo corpus went from\n0/10 to 10/10\n— zero usable data to fully usable data. That is not a gradient improvement. That is the difference between a corpus you can reason about and a corpus that produces no signal at all.\nWhat was actually broken\nThe field test exposed four harness failures. Each one looked like a model problem but was not.\nProblem\nWhat it looked like\nWhat it actually was\nduplicate rows\nmodel inconsistency\nsame-day result append contamination\nparse failure on valid-looking output\nweak model\nparser grabbed too much text\ncontext validation failure\nbad extraction\nblank\ncontext\nitems rejected too aggressively\nblocked raw runs\nmodel/runtime failure\nmissing input timestamps\nThis distinction matters because each of those failure modes points to a different fix path. If you treat all of them as \"the model is bad,\" you will optimize the wrong layer.\nA\ngolden\nresult set with 11 rows for a 10-trajectory corpus is not evidence of model instability. It is evidence that the reporting path allowed rerun contamination. Raw corpus failures caused by missing timestamps are not evidence that the LLM could not solve the task. They are evidence that the input contract was incomplete.\nEach misdiagnosis has a real cost. Misdiagnose duplicate rows as model instability and you spend weeks tuning temperature for a file-append bug. Misdiagnose parse failure as a weak model and you upgrade to a more expensive model for a greedy JSON slicer. Misdiagnose blocked raw runs as runtime failure and you debug the execution environment for missing timestamps in the input data.\nThe cost of misdiagnosis is not just wasted time. It is that you build confidence in the wrong mental model of your own system.\nThe fixes changed the meaning of the benchmark\nWe made five specific changes:\nreset\nresults.jsonl\nand\nsummary.json\nat run start\nextract the first balanced JSON object instead of slicing from first\n{\nto last\n}\nfilter blank\ncontext\nitems before validation\nstrengthen the prompt with a concrete filled JSON example\nrepair missing timestamps in blocked raw corpus assets\nThe report describes the transition in one sentence: the fixes moved the benchmark from\n\"mostly noise\" to \"mostly signal.\"\nThat is the single most important shift in the entire field test. Not because the numbers improved — but because the numbers started meaning something.\nHere is a concrete example. Before the fixes, the local Llama model on\ngolden\nproduced 3 parseable rows out of 11. After the fixes, it produced 10 out of 10, and the result was: 8 pass, 2 fail. That 8P/2F is a real signal about the product's extraction quality. The earlier 3/11 was not a signal about anything except the parser.\nThe raw corpora told the same story at a different scale\nThe curated corpora are the cleanest signal, but the raw corpora are where volume lives. And the raw data told the same story at much larger scale.\nRaw corpus\nBefore fixes\nAfter fixes\nraw/synthetic\n124 / 145 rows processed\n145 / 145\nraw/sibling-repos\n0 / 10 rows processed\n10 / 10\nraw/opencode\npartially blocked\nfully runnable\nraw/ci\npartially blocked\nfully runnable\nraw/corrections\npartially blocked\nfully runnable\nraw/cross-session\npartially blocked\nfully runnable\nSix raw corpora. Before the fixes, two were completely blocked and four were partially blocked. After the fixes, all six ran end to end. That is not \"we improved some metrics.\" That is \"we went from a benchmark that could not process its own data to one that could.\"\nThe report makes an observation about this that I think is underappreciated: a working runner plus a stable corpus is a\ncompounding asset\n. Each future iteration starts from the accumulated knowledge of the last one, instead of rediscovering the same benchmark setup problems every time. That is the difference between a benchmark that gets better with use and one that decays.\nThe transition from plumbing to judgment\nThere is a moment in every field test where the character of the work changes. For CauterRule, that moment was the parser fix.\nBefore the fix, the work was\nplumbing\n: file contamination, JSON slicing, blank context items, missing timestamps. Each problem had a clear cause and a clear fix. The fixes were satisfying — numbers jumped, blocked corpora opened up, the benchmark started working.\nAfter the fix, the work was\njudgment\n: is the trigger too broad? Is the directive specific enough? Does the replay logic correctly interpret the candidate? Does the product avoid extracting from cases where it should remain silent?\nPlumbing fixes produce dramatic gains. Judgment-quality improvements require more careful scoring, better matcher design, improved negative-case handling, and human-calibrated evaluation. The project is leaving the easy gains behind and entering the more important work.\nThat is a healthy transition. But it is also a warning: the next set of improvements will not come from a one-line parser fix. They will come from harder, slower, more deliberate work on the product's decision quality.\nWhat the benchmark started saying once it became honest\nOnce the harness stopped dominating the results, three conclusions became stable:\nthe product could extract many candidate rules from positive or obvious failure trajectories\nthe product was still much weaker on\nsuccesses\n,\nfailures/negative\n, and\nnearmiss\nthe main unresolved problem had shifted from formatting to judgment quality\nThe report frames this as the project's biggest win — not a benchmark score, but the moment when \"the remaining weak spots now point to the product, not to the test rig.\" Once that happened, the real product problem became visible for the first time.\nAnd here is the quote from the report that stuck with me most:\nA system that never produces useful output is obviously immature. A system that often produces useful-looking output, but still has weak safety boundaries, is more subtle and more dangerous. CauterRule is clearly in the second category.\nThe benchmark fixes did not make the product look good. They made the product look\nhonest\n. And honest meant: good at the easy half, visibly weak on the hard half, and dangerous in exactly the way that matters for trust.\nThe engineering foundation held while the benchmark broke\nOne detail from the report that deserves more attention: while the benchmark harness was manufacturing failure, the deterministic foundation was solid.\n844+ deterministic tests passing\n104/104 Docker tests passing\nexport/import validation passing\nredaction validation passing\ngold-family validation passing\nperformance baselines within threshold\nIt would have been easy to dismiss the whole system as broken. It was not. The application core — the serialization, the CLI, the Docker environment, the export formats, the redaction engine — was reliable. The noise was concentrated in the benchmark layer, not the product layer. If the substrate had also been shaky, the field test would have been unrecoverable.\nWhat I learned from this\nA few things that I think generalize beyond CauterRule:\nYour benchmark is a system too.\nIt has its own bugs, its own input contracts, its own failure modes. If you do not test your benchmark, you are trusting an untested system to judge a tested one.\nSignal loss is invisible until you measure it.\nWe did not know we were losing 73% of\ngolden\nsignal until we fixed the parser and saw the numbers jump. Before that, the lost signal looked like \"the model is bad.\" The failure was silent.\nFailure bucketing changes your roadmap.\nIf you lump all failures into \"model problem,\" you optimize the model. If you split them into model / parser / corpus / scoring, you discover that most of your early failures are not model problems at all — and your roadmap shifts.\nA plausible result from a broken benchmark is worse than no result.\nNo result forces you to investigate. A plausible result from a broken harness gives you false confidence. The 3/11\ngolden\nrows were plausible enough to interpret. They were wrong enough to mislead.\nFix the benchmark before you optimize the model.\nIf you optimize the model while the benchmark is broken, you are fitting to noise. Every model improvement you measure through a broken harness is suspect.\nRaw breadth without replay clarity is not yet evidence.\nThe raw corpora now all run end to end, but broad sets like\nraw/ci\nstill generate many inconclusive results. The product can\nprocess\nthem, but the benchmark does not yet turn that processing into consistently actionable quality signals.\nWhy this matters beyond one project\nThe broader lesson is not \"always trust local models more.\" It is narrower and more useful:\nnever let your benchmark failure masquerade as a model failure.\nIf you are building an agent system, memory system, or any kind of model-evaluated workflow, split failures into buckets before you interpret them:\nmodel problem\nparser/tool problem\ncorpus/input problem\nscoring/replay problem\nIf you do not, the benchmark can become the noisiest component in the system while still pretending to be the most authoritative one.\nThat is the part I would most want other engineers to argue with, test, or improve on. Not because it is abstract, but because once the benchmark becomes believable, the next round of product work becomes much more honest too.\nCauterRule v0.1.0 is released.\nThe full field test report — including the parser-fix before/after data, 844+ deterministic tests, 104 Docker tests, and model-by-model breakdowns — lives\nhere\n. The\nrepo\nis public. Install with\npip install cauterule\n.\nChangelog\n·\nRelease notes", "metadata": { "title": "When Your Benchmark Finally Tells the Truth - DEV Community", "description": "Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent... Tagged with ai, llm, agents, healthydebate.", "keywords": "ai, llm, agents, healthydebate, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/debashish_ghosal/when-your-benchmark-finally-tells-the-truth-534h", "og:title": "When Your Benchmark Finally Tells the Truth", "og:description": "Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmk15nwov85vbj1se6nc5.png", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "When Your Benchmark Finally Tells the Truth", "twitter:description": "Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmk15nwov85vbj1se6nc5.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://github.com/deghosal-2026/CauterRule", "label": "GitHub", "type": "a" }, { "href": "https://pypi.org/project/cauterule/", "label": "PyPI", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.1.0/FIELD_TEST_REPORT.md", "label": "field test report", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/releases/tag/v0.1.0", "label": "Release notes", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md", "label": "Changelog", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:50:20Z", "source_url": "https://dev.to/debashish_ghosal/when-your-benchmark-finally-tells-the-truth-534h", "papers": [], "actual_video_files": [], "body_chars": 11801, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/9be2da967e11/pages/source.html", "local_html": "attachments/9be2da967e11/pages/article.html", "body_base_path": "attachments/9be2da967e11/pages/", "body_markdown_path": "attachments/9be2da967e11/pages/article.md" }