{ "source_file": "0930", "source_platform": "Dev.to", "author": "Debashish Ghosal", "title_listed": "When Your Judge Can't Decide", "keywords": "", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://dev.to/debashish_ghosal/when-your-judge-cant-decide-1252", "publish_date": "2026-09-07", "corpus_stage": "", "notes": "", "id": "9d613d1aeba9", "collected_at": "2026-10-02T14:06:18", "title": "When Your Judge Can't Decide - DEV Community", "body_markdown": "Update — v0.1.0 released.\nCauterRule is now live on\nGitHub\nand\nPyPI\n. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.\npip install cauterule\ngives you the full CLI, MCP server, 7 export formats, and a bundled git rule pack. The\nfield test report\nevaluated 4 models across 394 trajectories and is the source for every number in this article.\nRelease notes\n·\nChangelog\nCauterRule\nis an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.\nEvery benchmark produces three buckets: pass, fail, and inconclusive. Most people only read the first two. In CauterRule's field test, the third bucket was bigger than both of them combined — and it was the one that mattered most.\nThe majority of our results were \"can't tell\"\nAcross all four models and the full corpus, the field test produced 1,538 candidates. Here is how they split:\nVerdict\nCount\n% of total\nPass\n365\n23.7%\nFail\n359\n23.3%\nInconclusive\n814\n52.9%\nMore than half of every result the benchmark produced was inconclusive. The replay engine could not decide whether the rule was good or bad. It did not pass the candidate. It did not reject it. It returned \"maybe.\"\nThat is not a fringe bucket. It is the majority of the output. And it is the bucket that most benchmark reports skip entirely, because it is uncomfortable to talk about. A pass is a win. A fail is a loss. An inconclusive is an embarrassment — it means your evaluation tool could not do its job.\nBut if you ignore the inconclusive bucket, you are ignoring more than half of your data. And you are probably drawing the wrong conclusions from the half you kept.\nThe inconclusive rate varied by model — but not the way you'd expect\nModel\nCandidates\nInconclusive\nInconclusive %\nLocal Llama 3.2B\n379\n189\n49.9%\nLocal Qwen 4B\n373\n209\n56.0%\nCloud GPT-4o-mini\n394\n248\n62.9%\nCloud Llama 3.1 8B\n392\n168\n42.9%\nThe strongest cloud model —\nmeta-llama/llama-3.1-8b-instruct\n— had the lowest inconclusive rate at 42.9%. That makes sense: better extraction produces more specific triggers, which are easier for the replay engine to verify.\nBut\ngpt-4o-mini\n— a paid cloud model — had the\nhighest\ninconclusive rate at 62.9%. Higher than both local models. This is the kind of finding that only surfaces when you stop reading pass counts and start reading the full distribution. On raw pass count, GPT-4o-mini looked comparable to the locals (77 vs 72 vs 93). On inconclusive rate, it was the most indecisive model in the field test.\nThat does not mean GPT-4o-mini is a bad model. It means the model produced candidates that the replay engine could not evaluate. The model's output was not worse — it was\nless scorable\n. Those are different failures, and they point to different fixes.\nWhat \"inconclusive\" actually means\nThe replay engine takes a candidate rule and runs it against reference trajectories. It checks whether the rule's trigger fires on the right cases (recall) and stays silent on the wrong ones (precision). If both precision and recall are high enough, the verdict is\npass\n. If the rule clearly fires on wrong cases or misses all right cases, the verdict is\nfail\n.\nIf neither condition is clearly met, the verdict is\ninconclusive\n.\nThe matcher that decides this is not a sophisticated judge. It uses substring and token-overlap heuristics. It checks whether words in the trigger appear in the trajectory. If the overlap is in a gray zone — not high enough to pass, not low enough to fail — the matcher returns inconclusive.\nThis is not a secret. The field test report names it explicitly as the #1 engineering target:\nReplay and matcher calibration is now the highest-value engineering target.\nThe report also says:\nThe matcher and simulator are still heuristic and can over- or under-fire on substring/token overlap.\nThat means the inconclusive bucket is contaminated. It contains at least two different populations:\nRules that are too vague to verify\n— the model extracted a generic trigger like \"when a command fails\" instead of \"when git push fails with non-fast-forward.\" The matcher cannot tell whether this fires on the right cases because the trigger is too broad to evaluate. This is a model problem.\nRules that are specific and reasonable but the matcher is too weak to confirm\n— the model extracted a good rule, but the substring/token-overlap heuristic cannot recognize that the trigger matches the reference trajectory. This is an engine problem.\nThe field test data cannot cleanly separate these two populations. That is itself a finding: the benchmark is not instrumented well enough to tell you\nwhy\na result is inconclusive. And without that, you cannot tell whether the inconclusive bucket is mostly model weakness or mostly engine weakness.\nThe curated vs raw split reveals where inconclusives concentrate\nThe sharpest insight comes from splitting the inconclusive rate by corpus type.\nModel\nCurated inconclusive %\nRaw inconclusive %\nLocal Llama 3.2B\n33.0%\n55.2%\nLocal Qwen 4B\n34.0%\n63.4%\nCloud GPT-4o-mini\n38.3%\n70.7%\nCloud Llama 3.1 8B\n14.9%\n51.7%\nOn curated corpora, the inconclusive rates are manageable — especially for the strongest cloud model (14.9%). On raw corpora, every model jumps above 50%. GPT-4o-mini hits 70.7%.\nTwo raw corpora drove most of the inconclusive volume:\nRaw corpus\nLlama 3.2B inc %\nQwen 4B inc %\nGPT-4o-mini inc %\nLlama 8B inc %\nraw/ci\n45.0%\n81.8%\n90.0%\n54.5%\nraw/sibling-repos\n90.0%\n90.0%\n70.0%\n100.0%\nraw/ci\nproduced 110 trajectories per model. For GPT-4o-mini, 99 of those 110 were inconclusive — 90%. For Qwen 4B, 90 of 110 — 82%. The replay engine was essentially paralyzed on this corpus.\nraw/sibling-repos\nwas worse: 10 trajectories per model, and for the strongest cloud model, all 10 were inconclusive. 100%. The engine could not produce a single decisive verdict.\nThe report's interpretation is direct:\nRaw corpora, especially\nraw/ci\n, produce many inconclusive outcomes because the current replay setup is a weak fit for that breadth.\nraw/sibling-repos\nremains mostly inconclusive for both cloud models, meaning harness-transfer quality is still not strong.\nThat is an important distinction. The inconclusive problem is not evenly distributed. It is concentrated in raw corpora where the replay engine's heuristics are a poor fit for the data shape. On curated corpora — where the trajectories are clean, the failure signatures are clear, and the expected rules are known — the engine performs much better.\nWhy better models did not solve this\nIf inconclusives were mostly a model problem, stronger models should reduce the inconclusive rate. They did, but only partially.\nModel\nInconclusive %\nPass %\nFail %\nLocal Llama 3.2B\n49.9%\n19.0%\n31.1%\nLocal Qwen 4B\n56.0%\n24.9%\n19.0%\nCloud GPT-4o-mini\n62.9%\n19.5%\n17.5%\nCloud Llama 3.1 8B\n42.9%\n31.4%\n25.8%\nThe strongest model reduced inconclusives from ~50-63% to ~43%. That is a real improvement, but it left the inconclusive bucket as the largest category. Even the best model in the field test could not get the replay engine to a decisive verdict on the majority of its output.\nAnd the cloud model with the best parse stability —\ngpt-4o-mini\n— had the\nworst\ninconclusive rate. Perfect formatting did not help the matcher decide. That confirms what the report says: better structured output does not automatically translate to good replay outcomes. The bottleneck is not extraction. It is judgment.\nThis is the same conclusion the field test report reaches from a different angle. The cloud runs \"proved that better structured output does not automatically translate to good safety behavior or high replay pass rates.\" The inconclusive data is the concrete evidence for that claim.\nWhat the inconclusive bucket costs you\nThe inconclusive bucket is not neutral. It has real costs that compound if you ignore it.\nIt makes your pass rate look lower than it is.\nIf 53% of your results are inconclusive, your pass rate is measured against a denominator that includes a large chunk of \"we don't know.\" The true pass rate — the rate at which the model produces good rules — is somewhere between the reported pass rate and pass-plus-inconclusive. For Llama 8B, that range is 31.4%–74.2%. That is a massive uncertainty band, and it means the pass count is a floor, not a ceiling.\nIt makes your fail rate look lower than it is.\nThe same logic applies to fails. Some inconclusives are rules that should have been rejected but the matcher could not prove it. Those are hidden failures — safety risks that do not show up in the fail count because the engine was too weak to catch them.\nIt blocks promotion.\nIn a production system, inconclusive rules are not promoted. They sit in a queue waiting for human review or better evaluation. If 53% of your output is inconclusive, your promotion pipeline is choked. You are producing candidates faster than you can adjudicate them.\nIt hides the real product problem.\nWhen you cannot tell whether a rule is good or bad, you cannot tell whether your product is good or bad. The inconclusive bucket is a measurement gap, and measurement gaps are where product risks hide. The report is explicit: \"the benchmark does not yet turn that processing into consistently actionable quality signals.\"\nWhat a better judge would look like\nThe current replay engine is a string matcher. It checks whether words in the trigger appear in the trajectory. That is a reasonable starting point, but it is not sufficient for the problem CauterRule is trying to solve.\nA better judge would need to do three things the current engine cannot:\nSemantic matching.\nInstead of substring overlap, the matcher needs to understand whether a trigger like \"when a non-fast-forward push is rejected\" fires on a trajectory where the error message says \"Updates were rejected because the remote contains work that you do not have locally.\" Those are the same event described differently. A string matcher misses that. A semantic matcher — even a lightweight embedding-based one — would catch it.\nCorpus-aware calibration.\nThe same matcher threshold does not work for all corpora. Curated corpora with clean failure signatures can tolerate stricter matching. Raw corpora with messy, multi-step trajectories need looser matching or a different matching strategy entirely. The report says the replay setup is \"a weak fit for that breadth.\" The fix is not one better matcher. It is a matcher that adapts to the corpus type.\nInconclusive attribution.\nWhen the matcher returns inconclusive, it should say why. Is the trigger too broad? Is the trajectory too messy? Is the matcher's vocabulary too narrow? Without attribution, every inconclusive looks the same, and you cannot prioritize fixes. The current benchmark lumps all inconclusives together, which is why the report says \"the benchmark still mixes several failure modes together.\"\nWhat I learned from this\nThe inconclusive bucket is the most important column in the table.\nPass and fail tell you what the system decided. Inconclusive tells you what the system\ncould not\ndecide. In a product where the core value proposition is \"we promote only safe rules,\" the inability to decide is a bigger risk than a wrong decision. A wrong decision can be caught and reverted. An undecidable result sits in limbo forever.\nYour benchmark is only as good as its judge.\nCauterRule's extraction layer improved dramatically after the parser fixes. The model comparison produced clean, parseable candidates across 1,538 rows. But the replay engine — the thing that decides whether those candidates are good — is still a heuristic string matcher. That means the benchmark's quality ceiling is set by the judge, not by the models. Investing in better models while the judge is weak is optimizing the wrong layer.\nRaw breadth without evaluation depth is not evidence.\nThe raw corpora produced 1,175 candidates across all models. 745 of those were inconclusive — 63%. The product can\nprocess\nraw corpora, but it cannot\nevaluate\nthem. The report says this directly: \"broad coverage that produces mostly inconclusives is not yet the strongest evidence of product trustworthiness.\" If you are benchmarking a learning system, do not confuse processing coverage with evaluation coverage.\nThe inconclusive rate is a product health metric, not just a benchmark metric.\nIf your inconclusive rate is 53%, your product has a measurement problem that blocks every downstream decision — promotion, safety, trust. Tracking the inconclusive rate over time tells you whether the product is getting better at judgment, not just at extraction. It should be a first-class metric in any learning system's dashboard.\nOpen questions\nWould a semantic matcher (even a lightweight embedding-based one) materially reduce the inconclusive rate on raw corpora, or is the problem deeper than matching strategy? The current data cannot answer this because we only tested one matcher.\nThe curated inconclusive rate for Llama 8B was 14.9%. Is that the realistic floor for the current matcher, or can calibration push it lower? If 15% is the floor, that means even on clean data, 1 in 7 rules cannot be evaluated.\nGPT-4o-mini had the highest inconclusive rate (62.9%) despite perfect parse stability. Is that because the model produces more nuanced triggers that the string matcher cannot handle, or because it produces vaguer triggers? The data does not distinguish.\nThe\nraw/sibling-repos\ncorpus produced 100% inconclusives for the strongest cloud model. Is that a matcher problem, a corpus problem, or a harness-transfer problem? The report says \"harness-transfer quality is still not strong,\" which suggests the corpus itself may be the issue, not the engine.\nIf we split the inconclusive bucket by root cause (model vagueness vs engine weakness), what proportion would fall into each? That split would completely change the product roadmap. If 70% of inconclusives are engine weakness, the fix is replay calibration. If 70% are model vagueness, the fix is prompt engineering and trigger specificity.\nThe broader lesson\nIf you are building any system that evaluates AI output — a replay engine, a guardrail checker, a quality scorer, a promotion gate — track the inconclusive rate as seriously as you track pass and fail. The inconclusive bucket is where measurement gaps live. And measurement gaps are where product risks hide.\nA benchmark that returns 53% \"can't tell\" is not measuring quality. It is measuring uncertainty. And uncertainty is the most expensive thing in a learning system, because it blocks every downstream decision without telling you why.\nThe next milestone for CauterRule is not more extraction breadth or better models. It is a judge that can decide.\nCauterRule v0.1.0 is released.\nThe full model-by-model breakdown — including pass/inconclusive/fail distributions for all 4 models across curated and raw corpora — is in the\nfield test report\n. The\nrepo\nis public. Install with\npip install cauterule\n.\nChangelog\n·\nRelease notes", "metadata": { "title": "When Your Judge Can't Decide - DEV Community", "description": "Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent... Tagged with ai, llm, testing, agents.", "keywords": "ai, llm, testing, agents, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/debashish_ghosal/when-your-judge-cant-decide-1252", "og:title": "When Your Judge Can't Decide", "og:description": "Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5v4t1qj8eqgtb097z423.png", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "When Your Judge Can't Decide", "twitter:description": "Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5v4t1qj8eqgtb097z423.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://github.com/deghosal-2026/CauterRule", "label": "GitHub", "type": "a" }, { "href": "https://pypi.org/project/cauterule/", "label": "PyPI", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.1.0/FIELD_TEST_REPORT.md", "label": "field test report", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/releases/tag/v0.1.0", "label": "Release notes", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md", "label": "Changelog", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:50:19Z", "source_url": "https://dev.to/debashish_ghosal/when-your-judge-cant-decide-1252", "papers": [], "actual_video_files": [], "body_chars": 15064, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/9d613d1aeba9/pages/source.html", "local_html": "attachments/9d613d1aeba9/pages/article.html", "body_base_path": "attachments/9d613d1aeba9/pages/", "body_markdown_path": "attachments/9d613d1aeba9/pages/article.md" }