{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "Killed by the Word 'git': One Token of Coincidence, 40 Points of Pass Rate",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/killed-by-the-word-git-one-token-of-coincidence-40-points-of-pass-rate-140f",
  "publish_date": "2026-09-14",
  "corpus_stage": "",
  "notes": "",
  "id": "efd827c9332a",
  "collected_at": "2026-10-02T14:06:06",
  "title": "Killed by the Word 'git': One Token of Coincidence, 40 Points of Pass Rate - DEV Community",
  "body_markdown": "Update — v0.3.1 released.\nCauterRule is now live on\nGitHub\nand\nPyPI\n. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.\npip install cauterule\ngives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The\nv0.3.1 field test report\nevaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the source for every number below.\nRelease notes\n·\nChangelog\nCauterRule\nis an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.\nThe article that split extraction from replay ended on a worked example:\nF-001\n, the git case. The model extracted the right rule —\n\"when git push fails with non-fast-forward, pull latest changes before pushing\"\n— almost verbatim. Replay scored it\nprevented 5, broken 3\n, and the verdict was INCONCLUSIVE. The three \"broken\" successes were\nS-022-git-pull\n,\nS-023-git-status\n, and\nNM-044-git-commit-hook\n. A\nsuccessful\ngit status\ncounted as broken by a git-push rule because the two share the token\ngit\n.\nThat was the diagnosis. This is the fix, and the 40+ points it bought.\nThe bug: \"broken\" counted coincidence as interference\nThe scorer's job is to decide whether a rule does net good: does it prevent more failures than it breaks successes? The\nbroken\ncount is supposed to mean\n\"this rule would have fired when things were working\"\n— genuine interference. But \"would have fired\" was computed with the same lexical matcher as everything else. Sharing a token\nis\na match, so:\nevery success that merely\nrhymes\nwith the rule counted as a success the rule\nbroke\n.\nFor a domain-heavy corpus like\nfailures/positive\n, where git rules sit next to git successes, that is not a rare edge case. It is the dominant failure mode. Combined with a\nprecision < 0.5\nbar and a zero-tolerance near-miss penalty, the result was\nfailures/positive\nat\n8–10% pass\nfor a release: correct rules, systematically demoted by their own neighbors.\nThe fix (#723 + #724): a margin, an ordering, and a definition\nDomain-gate\nbroken\n.\nA success can only break a rule if it is in the rule's own domain. Cross-domain coincidences stop counting outright.\nRequire a match-strength margin.\nA same-domain success counts as broken only if it clears\nthreshold + 0.10\n. The one-sentence version of the principle:\na near-threshold success match is coincidence, not interference.\nA\ngit status\nsuccess that only weakly matches a push-specific trigger is no longer a \"break.\" It is two git commands sharing vocabulary, which is what they are.\nReorder the scorer and bound the near-miss band (#724).\nThe old ordering let the zero-tolerance near-miss penalty over-fire on rules that prevented more failures than they broke. The new order: no-signal → inconclusive;\nbroken > prevented\n→ fail;\nnear_misses > 2\n→ inconclusive; otherwise → pass. The definitional shift:\n\"breaks fewer successes than it prevents failures\"\nis the definition of a useful rule, and the scorer now admits net-positive rules (precision ≥ 0.5) instead of downgrading them.\nTwo supporting fixes made the candidate pool honest while we were in the neighborhood: recall-weighted ranking instead of precision-first (#731), and signature-aware 2-pass dedup — roughly\n40% of the two-pass candidates were identical\n, so the runner's \"best\" candidate disagreed with what production would have picked (#732).\nThe data\nfailures/positive\n(n=50), both cloud models:\nv0.3.0\nv0.3.1 gpt-4o-mini\nv0.3.1 llama-3.1-8b\nPass rate\n8% / 10%\n50% [0.37, 0.63]\n52% [0.39, 0.65]\nPrecision\n—\n0.544\n0.554\nRecall\n—\n0.221\n0.240\nGpt verdict breakdown on the corpus:\npass 25, no_signal 17, blocked_by_broken 3, blocked_by_near_miss 4, min_sample 1\n. The residual ~44% inconclusive is mostly\nno_signal\n, and the report attributes it to the same reference-pool precision limit we explicitly\naccepted\non\nraw/synthetic\n(J12): correct triggers each breaking 1–4\ngeneric same-domain\nsuccesses. Widening the margin further to rescue those would loosen a real safety trade-off, so we documented the limit instead of tuning past it.\nWhether\nF-001\n's specific candidate now passes lives in the committed per-trajectory artifacts; the corpus-level number above is the claim I am making, and it is the one with a confidence interval.\nThe proof the other side of the gate still held\nLoosening a positive gate is only a win if the negative gate holds, and v0.3.1 made that a measured claim instead of an aspiration:\nSafety measure\nv0.3.1 result\nnearmiss\n(rejection corpus)\n27/50 gate-silenced by recovery detection; of 23 active,\n0 accepted\n— Wilson 95% CI\n[0.857, 1.000]\nadversarial/*\n(9 corpora)\n0 accepted\nacross injection, misleading, contradiction, unsafe, poisoning, tool_output_injection, compounding_multiturn, unsafe_realistic, and both harmbench slices — both models\nGeneric triggers\n0.4% (gpt) / 0.7% (llama)\nvs <10% target\nTotal pass volume\n116/119 → 601/626\n— ~5×, with safety flat\nThree structural pieces make that table mean something:\nJ1 — rejection corpora are scored as rejections.\nnearmiss\nand\nadversarial/*\npass iff\naccepted == 0\n. The zero is a first-class measured quantity with a CI, not the absence of a logged failure. (v0.3.0 scored these through the extraction branch — an inverted gate that made 1 false accept read as a pass. The fix is one commit; the lesson is that \"pass\" must be defined per corpus type.)\nThe source-trust gate left the test harness.\nv0.3.0's #727 documented an adversarial trust gap as\ntest-only\n. #775/#776 wired the source-trust check into\nproduction auto-promotion\n— a rule whose evidence traces to an untrusted source is rejected before it can be promoted, in the product, not just in the field test.\nThe shipped container agrees.\nThe Docker field test went 159 →\n180/180\n(+21 new regression tests covering the 43 M2 code-review fixes, #762–#804), so the scorer, gate, and trust checks hold in the deployment artifact, not just the dev tree.\nWhat worked\nThe margin principle.\n\"Coincidence, not interference\" is one sentence, one constant (\n0.10\n), and worth most of the 40 points. Cheap fixes that encode a real distinction beat expensive fixes that encode a threshold.\nDefining \"useful\" before scoring.\nNet-positive — prevents more than it breaks — became the promotion standard the scorer implements. When the definition and the scorer agree, the scorer can be simple.\nZeros with CIs.\n0/23 accepted with a Wilson lower bound of 0.857 is a claim. 0/23 accepted with no denominator is a hope.\nLoosening one side while measuring the other.\nThe pass-rate jump and the safety table are the same release. Reporting both is what makes either of them credible.\nWhat didn't work\nToken overlap as interference.\nThe original\nbroken\ncount treated lexical match as causal. Two texts sharing a word is a fact about the texts; \"this rule broke this success\" is a claim about events. The scorer was grading the first and reporting the second.\nZero-tolerance near-miss penalty.\nIt over-fired on exactly the rules worth having — the ones that prevent more than they break. Bounding the band at\n> 2\nrestored the distinction.\nPrecision-first ranking.\nIt favored low-recall rules and starved the corpus of the specific, high-recall ones. Ranking should optimize what you promote.\nQuestions we still can't answer\nIs 0.10 the right margin, or should it be per-domain?\nGit domains are token-dense; browser-tool domains may not be. We have one margin and no curve.\nCan \"0 accepted\" be claimed where the model goes silent?\nJ16: llama-3.1-8b returned\nzero candidates\non\nunsafe_realistic\n13/20,\ncontradiction_harmbench\n10/15, and\nmisleading_harmbench\n4/15. Those zeros are lower bounds — silence is not rejection. The gate is safe either way, but the report must not dress a non-answer as a measured rejection.\nThe two protocol metrics we still haven't run.\nCross-session repeat-failure reduction (#741) and human-vs-replay agreement (#742): tooling complete, protocols not run. They are the difference between \"core gate closed\" and \"full gate closed,\" and neither is a blocker for the core promotion path.\nWhere does J12 stop?\nThe accepted\nraw/synthetic\nprecision limit is real, but \"accepted\" is a boundary we drew. Re-drawing it is a safety trade-off decision, not an engineering one — and it hasn't been made by anyone with more information than us.\nWhat I learned\nOverlap is not interference.\nA relation between two pieces of text is not a relation between two events. If your scorer counts \"shares a token\" as \"broke,\" add a margin or a domain gate — demand\nmore than coincidence\nbefore counting a collision.\nLoosening a gate is testable, and the test is the negative corpus.\nThe 40 points we gained on\nfailures/positive\nare only news if\nnearmiss\nand the nine adversarial corpora say zero afterwards. Run them in the same sweep, and publish both tables in the same report.\n\"Net positive\" is a definition worth stealing.\nA rule that breaks fewer successes than it prevents failures is useful, full stop. Most scorers I have seen implement some muddle of precision and recall instead of that sentence. The sentence is simpler and it matches the intent.\nReport zeros with their denominators and their CIs.\n0/23, [0.857, 1.000]. The lower bound is the honest number, and it is the number a skeptical reader will recompute.\nThe broader lesson\nWhen a gate kills your good outputs, find the single relation it is mis-grading — here, token overlap masquerading as interference — and replace it with a relation that costs almost nothing to compute: same domain, plus a margin. Then prove the other side of the gate with a measured zero, not a quiet one.\nThat closes the v0.3.1 arc: an honest ruler, a full haystack, a reachable floor, and a judge that finally distinguishes coincidence from interference. What's left is the work the report names but the release does not: the 0-accepted corpora, the cross-session protocol, and the human-agreement run — the difference between a gate we closed and a gate we could close.\nReferences\nCauterRule v0.3.1 release notes\nv0.3.1 field test report (Fix 2 + Fix 3, §7 safety metrics, Appendix A J1/J12/J16)\nDocker field test results (180/180, +21 regression tests)\nSECURITY.md (source-trust gate, #727)\nv0.3.0 field test report (the F-001 diagnosis)\nUser guide\n·\nChangelog\nCauterRule v0.3.1 is released.\nThe scorer before/after, the safety CIs, and the J1/J12/J16 issue journal are in the\nfield test report\n. The\nrepo\nis public. Install with\npip install cauterule\n.\nChangelog\n·\nRelease notes",
  "metadata": {
    "title": "Killed by the Word 'git': One Token of Coincidence, 40 Points of Pass Rate - DEV Community",
    "description": "Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent... Tagged with ai, llm, agents, testing.",
    "keywords": "ai, llm, agents, testing, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/killed-by-the-word-git-one-token-of-coincidence-40-points-of-pass-rate-140f",
    "og:title": "Killed by the Word 'git': One Token of Coincidence, 40 Points of Pass Rate",
    "og:description": "Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Firu4z24q0z0i1smjyv6n.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Killed by the Word 'git': One Token of Coincidence, 40 Points of Pass Rate",
    "twitter:description": "Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Firu4z24q0z0i1smjyv6n.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/deghosal-2026/CauterRule",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://pypi.org/project/cauterule/",
      "label": "PyPI",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.1/FIELD_TEST_REPORT.md",
      "label": "v0.3.1 field test report",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/release/v0.3.1/release-notes.md",
      "label": "Release notes",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md",
      "label": "Changelog",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.1/docker-test-results.md",
      "label": "Docker field test results (180/180, +21 regression tests)",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/SECURITY.md",
      "label": "SECURITY.md (source-trust gate, #727)",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md",
      "label": "v0.3.0 field test report (the F-001 diagnosis)",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/USER_GUIDE.md",
      "label": "User guide",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:15Z",
  "source_url": "https://dev.to/debashish_ghosal/killed-by-the-word-git-one-token-of-coincidence-40-points-of-pass-rate-140f",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 10615,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/efd827c9332a/pages/source.html",
  "local_html": "attachments/efd827c9332a/pages/article.html",
  "body_base_path": "attachments/efd827c9332a/pages/",
  "body_markdown_path": "attachments/efd827c9332a/pages/article.md"
}