{ "source_file": "0930", "source_platform": "Dev.to", "author": "Debashish Ghosal", "title_listed": "The Model Wrote the Right Rule and My Replay Rejected It: The Extraction-vs-Replay Split", "keywords": "", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://dev.to/debashish_ghosal/the-model-wrote-the-right-rule-and-my-replay-rejected-it-the-extraction-vs-replay-split-4304", "publish_date": "2026-09-12", "corpus_stage": "", "notes": "", "id": "22d3148c263d", "collected_at": "2026-10-02T14:06:11", "title": "The Model Wrote the Right Rule and My Replay Rejected It: The Extraction-vs-Replay Split - DEV Community", "body_markdown": "Update — v0.3.0 released.\nCauterRule is now live on\nGitHub\nand\nPyPI\n. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.\npip install cauterule\ngives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The\nv0.3.0 field test report\nevaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below.\nRelease notes\n·\nChangelog\nCauterRule\nis an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.\nFor two releases we treated \"the pass rate is low\" as one problem and reached for the extractor. It turned out to be two problems we had fused into a single number. This is the split, and it changed where we point next.\nThe assumption: one metric, two questions\nThe pipeline has two halves that answer two different questions:\nExtraction:\ngiven a failure, does the model produce the\nright\nrule?\nReplay / evaluation:\ngiven a rule, can we\nverify\nit against history?\nWe measured only the second, and we read its score as a verdict on the first. The replay gate that decides promotion computes:\nprecision\n=\nprevented\n/\n(\nprevented\n+\nbroken\n)\nrecall\n=\nprevented\n/\ntotal_failures\nEnter fullscreen mode\nExit fullscreen mode\nwhere\nprevented\nand\nbroken\ncome from\nsimulate()\n, which calls\nrule_matches()\n— a\ntext matcher\n. So\nprevented\nmeans \"the trigger's\nprose\nreached a similarity threshold against a reference failure's prose,\" and\nbroken\nmeans \"…against a reference success's prose.\" Nothing in that path asks whether applying the rule's\ndirective\nwould have changed the trajectory's outcome. The gate is a lexical resemblance check wearing a validation costume.\nThe F-001 example: the rule was right, the grader said no\nIn the\nfailures/positive\ncorpus, trajectory\nF-001\nis the canonical git case:\nexpected_rule\n(ground truth)\n\"when git push fails with non-fast-forward, pull latest changes before pushing\"\nextracted\nwhen\n\"when git push fails with non-fast-forward\"\nextracted\ndo\n\"pull latest changes before pushing\"\nThe model reproduced the reference rule almost verbatim. Extraction did its job. Then replay scored it:\nfailures_prevented = 5 successes_broken = 3 near_misses = 1\nprecision = 0.625 recall = 0.625 verdict = INCONCLUSIVE\nEnter fullscreen mode\nExit fullscreen mode\nThree \"broken\" successes were\nS-022-git-pull\n,\nS-023-git-status\n, and\nNM-044-git-commit-hook\n— a\nsuccessful\ngit status\ncounted as broken by a git-push rule, because the two share the token\ngit\n. A correct rule was demoted to inconclusive by three unrelated successes it merely rhymes with. The model wrote the right rule. The grader rejected it on the prose.\nThe assumption we made (and where it broke)\nAssumption:\nthe replay verdict tells us whether the extracted rule is good.\nIt tells us whether the extracted\ntrigger's surface form\nresembles stored reference\nsurface forms\n. Those are different claims. A correct rule phrased differently scores low; a wrong rule that happens to share vocabulary scores high. The metric is a\ntext-similarity proxy\n, and it is the gate.\nThe telling part: the undercount is not unique to replay. The corpus carries a\nground-truth rule\n(\nexpected_rule\n) on 23/50\nfailures/positive\ntrajectories and 288\nreference-expansion\ntrajectories — a ready-made extraction-accuracy instrument we were never scoring against. When we do score it naively (token-F1 of the extracted rule vs\nexpected_rule\n), we get only ~\n0.58\n(llama-3.1-8b) / ~\n0.50\n(gpt-4o-mini). Not because the model is wrong — F-001 is near-verbatim — but because it\nrewords\n, and a token comparator can't see through rewording. The same paraphrase gap that breaks replay also undercounts extraction. We had one brittle text-matching lens in two places.\nThe data\nCloud models,\nfailures/positive\n(23 trajectories carry\nexpected_rule\n):\nMetric\ngpt-4o-mini\nllama-3.1-8b\nWhat it measures\nReplay pass rate\n8%\n10%\ntrigger prose vs reference prose\nExtraction token-F1 vs\nexpected_rule\n0.50\n0.58\nextracted rule vs ground truth (text)\nF-001 extracted vs\nexpected_rule\nnear-verbatim\nnear-verbatim\na correct rule\nF-001 replay verdict\ninconclusive\ninconclusive\nthe grader, not the rule\nThe two left-column numbers look bad and point at the model. The right two columns say the model was fine and the\nmeasurements\nare the weak link. The spread is the whole story: extraction is substantively right but lexically variable, and every metric we ship is lexical.\nWhat worked\nNaming the two halves separately stopped the misattribution.\n\"Low pass rate\" is a symptom; \"extraction accuracy\" and \"replay fidelity\" are the two variables. Once separated, the model stops taking the blame for the grader.\nThe ground truth was already in the corpus.\nexpected_rule\nneeds no new labeling on\nfailures/positive\n— it was being parsed out and dropped. Picking it up is the cheapest high-signal metric we have.\nOne worked example did more than any average.\nF-001 — right rule, inconclusive verdict, three token-sharing successes — is the argument in one line. Averages hid it; the single case exposed the mechanism.\nWhat didn't work\nWe had a ground-truth rule and never scored against it.\nexpected_rule\nis dropped at parse (\nTrajectory\nhas no such field), so the only quality number was the replay verdict. We graded the homework with the wrong rubric for a full release.\nThe replay metric masquerades as validation.\n\"Tested against history before promotion\" is the product's core promise, but the test is lexical resemblance. A reader who believes the promise and inspects the gate finds prose matching.\nToken-F1 undercounts extraction too.\nReaching only ~0.5–0.6 against known-good rules, a token-F1 extraction score would\nalso\nwrongly suggest the model is mediocre. The comparator needs a semantic or signature signal, or it repeats the replay's error on the extraction side.\nBoth metrics share one brittle primitive.\nBecause replay and the naive extraction check both lean on the same token matcher, fixing one without the other leaves the split half-measured.\nQuestions we still can't answer\nShould replay mean \"does this rule\nprevent this class\nof failure\" (behavioral) or \"does it\nmatch this history\n\" (lexical)? We have been shipping the second and calling it the first.\nCan we validate by\napplying\nthe directive to the reference trajectory and checking the outcome flips — turning\nprevented\nfrom \"text matched\" into \"outcome changed\"?\nIs\nexpected_rule\ndense enough to backfill onto\ngolden\n(currently null) so extraction quality has a clean per-corpus number everywhere?\nAt what point does a rule that is right-but-reworded deserve credit, and who decides — a threshold, an embedding floor, or a human?\nWhat I learned\nA pass rate is a composite; decompose it before you optimize it.\n\"8–10% pass\" fused extraction and evaluation. The moment we split the number, the model exonerated itself and the grader took the hit.\nGround truth you don't score against is a decoration.\nThe corpus already knew the answer (\nexpected_rule\n). Not wiring it into a metric meant we optimized the wrong half while the right half went unmeasured.\nIf your \"validation\" only reads words, it can't validate meaning.\nThe replay gate grades whether the trigger's prose rhymes with stored prose. That is a retrieval property, not a correctness property — and it is the gate.\nOne crisp failure case beats a dozen averages.\nF-001 is the entire diagnosis. When a metric makes a known-good rule fail, the metric is the bug, full stop.\nThe broader lesson\nIf you build a loop that\nproduces\nan artifact and then\nscores\nit, keep the two measurements separate and keep them honest. A score that grades surface form instead of behavior will make good outputs look bad and bad outputs that share vocabulary look fine — in both halves of the loop at once.\nCauterRule spent two releases tuning an extractor that was already writing the right rule, because the number it was chasing was really a prose-similarity score. The fix is not a better model. It is (a) scoring extraction against the\nexpected_rule\nwe already have, with a comparator that can see past rewording, and (b) making the replay gate check whether the\ndirective\nchanges the outcome, not whether the\ntrigger\nrhymes. Same model. Same corpus. Two honest metrics instead of one flattering lie.\nReferences\nCauterRule v0.3.0 release notes\nv0.3.0 field test report\nCorpus format spec (\nexpected_rule\n, domain labels)\nPRD — success metrics\nPRD — architecture\nUser guide\n·\nChangelog\nCauterRule v0.3.0 is released.\nThe replay and extraction data are in the\nfield test report\n. The\nrepo\nis public. Install with\npip install cauterule\n.\nChangelog\n·\nRelease notes", "metadata": { "title": "The Model Wrote the Right Rule and My Replay Rejected It: The Extraction-vs-Replay Split - DEV Community", "description": "Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent... Tagged with ai, replay, agents, testing.", "keywords": "ai, replay, agents, testing, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/debashish_ghosal/the-model-wrote-the-right-rule-and-my-replay-rejected-it-the-extraction-vs-replay-split-4304", "og:title": "The Model Wrote the Right Rule and My Replay Rejected It: The Extraction-vs-Replay Split", "og:description": "Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F130v3cb62bzks15sywje.png", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "The Model Wrote the Right Rule and My Replay Rejected It: The Extraction-vs-Replay Split", "twitter:description": "Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F130v3cb62bzks15sywje.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/22d3148c263d/code/body-000.txt", "chars": 84, "language": "text" }, { "file": "attachments/22d3148c263d/code/body-001.txt", "chars": 135, "language": "text" } ], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://github.com/deghosal-2026/CauterRule", "label": "GitHub", "type": "a" }, { "href": "https://pypi.org/project/cauterule/", "label": "PyPI", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md", "label": "v0.3.0 field test report", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/release/v0.3.0/release-notes.md", "label": "Release notes", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md", "label": "Changelog", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/corpus/format-spec.md", "label": "Corpus format spec ( expected_rule , domain labels)", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/design/prd/07-success-metrics.md", "label": "PRD — success metrics", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/design/prd/02-architecture.md", "label": "PRD — architecture", "type": "a" }, { "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/USER_GUIDE.md", "label": "User guide", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:50:16Z", "source_url": "https://dev.to/debashish_ghosal/the-model-wrote-the-right-rule-and-my-replay-rejected-it-the-extraction-vs-replay-split-4304", "papers": [], "actual_video_files": [], "body_chars": 8812, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/22d3148c263d/pages/source.html", "local_html": "attachments/22d3148c263d/pages/article.html", "body_base_path": "attachments/22d3148c263d/pages/", "body_markdown_path": "attachments/22d3148c263d/pages/article.md" }