Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.
pip install cauterulegives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.1 field test report evaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the source for every number below. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
The article that split extraction from replay ended on a worked example: F-001, the git case. The model extracted the right rule — "when git push fails with non-fast-forward, pull latest changes before pushing" — almost verbatim. Replay scored it prevented 5, broken 3, and the verdict was INCONCLUSIVE. The three "broken" successes were S-022-git-pull, S-023-git-status, and NM-044-git-commit-hook. A successful git status counted as broken by a git-push rule because the two share the token git.
That was the diagnosis. This is the fix, and the 40+ points it bought.
The scorer's job is to decide whether a rule does net good: does it prevent more failures than it breaks successes? The broken count is supposed to mean "this rule would have fired when things were working" — genuine interference. But "would have fired" was computed with the same lexical matcher as everything else. Sharing a token is a match, so:
every success that merely rhymes with the rule counted as a success the rule broke.
For a domain-heavy corpus like failures/positive, where git rules sit next to git successes, that is not a rare edge case. It is the dominant failure mode. Combined with a precision < 0.5 bar and a zero-tolerance near-miss penalty, the result was failures/positive at 8–10% pass for a release: correct rules, systematically demoted by their own neighbors.
Domain-gate broken. A success can only break a rule if it is in the rule's own domain. Cross-domain coincidences stop counting outright.
Require a match-strength margin. A same-domain success counts as broken only if it clears threshold + 0.10. The one-sentence version of the principle:
a near-threshold success match is coincidence, not interference.
A git status success that only weakly matches a push-specific trigger is no longer a "break." It is two git commands sharing vocabulary, which is what they are.
Reorder the scorer and bound the near-miss band (#724). The old ordering let the zero-tolerance near-miss penalty over-fire on rules that prevented more failures than they broke. The new order: no-signal → inconclusive; broken > prevented → fail; near_misses > 2 → inconclusive; otherwise → pass. The definitional shift: "breaks fewer successes than it prevents failures" is the definition of a useful rule, and the scorer now admits net-positive rules (precision ≥ 0.5) instead of downgrading them.
Two supporting fixes made the candidate pool honest while we were in the neighborhood: recall-weighted ranking instead of precision-first (#731), and signature-aware 2-pass dedup — roughly 40% of the two-pass candidates were identical, so the runner's "best" candidate disagreed with what production would have picked (#732).
failures/positive (n=50), both cloud models:
| v0.3.0 | v0.3.1 gpt-4o-mini | v0.3.1 llama-3.1-8b | |
|---|---|---|---|
| Pass rate | 8% / 10% | 50% [0.37, 0.63] | 52% [0.39, 0.65] |
| Precision | — | 0.544 | 0.554 |
| Recall | — | 0.221 | 0.240 |
Gpt verdict breakdown on the corpus: pass 25, no_signal 17, blocked_by_broken 3, blocked_by_near_miss 4, min_sample 1. The residual ~44% inconclusive is mostly no_signal, and the report attributes it to the same reference-pool precision limit we explicitly accepted on raw/synthetic (J12): correct triggers each breaking 1–4 generic same-domain successes. Widening the margin further to rescue those would loosen a real safety trade-off, so we documented the limit instead of tuning past it.
Whether F-001's specific candidate now passes lives in the committed per-trajectory artifacts; the corpus-level number above is the claim I am making, and it is the one with a confidence interval.
Loosening a positive gate is only a win if the negative gate holds, and v0.3.1 made that a measured claim instead of an aspiration:
| Safety measure | v0.3.1 result |
|---|---|
nearmiss (rejection corpus) |
27/50 gate-silenced by recovery detection; of 23 active, 0 accepted — Wilson 95% CI [0.857, 1.000] |
adversarial/* (9 corpora) |
0 accepted across injection, misleading, contradiction, unsafe, poisoning, tool_output_injection, compounding_multiturn, unsafe_realistic, and both harmbench slices — both models |
| Generic triggers | 0.4% (gpt) / 0.7% (llama) vs <10% target |
| Total pass volume | 116/119 → 601/626 — ~5×, with safety flat |
Three structural pieces make that table mean something:
nearmiss and adversarial/* pass iff accepted == 0. The zero is a first-class measured quantity with a CI, not the absence of a logged failure. (v0.3.0 scored these through the extraction branch — an inverted gate that made 1 false accept read as a pass. The fix is one commit; the lesson is that "pass" must be defined per corpus type.)0.10), and worth most of the 40 points. Cheap fixes that encode a real distinction beat expensive fixes that encode a threshold.broken count treated lexical match as causal. Two texts sharing a word is a fact about the texts; "this rule broke this success" is a claim about events. The scorer was grading the first and reporting the second.> 2 restored the distinction.unsafe_realistic 13/20, contradiction_harmbench 10/15, and misleading_harmbench 4/15. Those zeros are lower bounds — silence is not rejection. The gate is safe either way, but the report must not dress a non-answer as a measured rejection.raw/synthetic precision limit is real, but "accepted" is a boundary we drew. Re-drawing it is a safety trade-off decision, not an engineering one — and it hasn't been made by anyone with more information than us.Overlap is not interference. A relation between two pieces of text is not a relation between two events. If your scorer counts "shares a token" as "broke," add a margin or a domain gate — demand more than coincidence before counting a collision.
Loosening a gate is testable, and the test is the negative corpus. The 40 points we gained on failures/positive are only news if nearmiss and the nine adversarial corpora say zero afterwards. Run them in the same sweep, and publish both tables in the same report.
"Net positive" is a definition worth stealing. A rule that breaks fewer successes than it prevents failures is useful, full stop. Most scorers I have seen implement some muddle of precision and recall instead of that sentence. The sentence is simpler and it matches the intent.
Report zeros with their denominators and their CIs. 0/23, [0.857, 1.000]. The lower bound is the honest number, and it is the number a skeptical reader will recompute.
When a gate kills your good outputs, find the single relation it is mis-grading — here, token overlap masquerading as interference — and replace it with a relation that costs almost nothing to compute: same domain, plus a margin. Then prove the other side of the gate with a measured zero, not a quiet one.
That closes the v0.3.1 arc: an honest ruler, a full haystack, a reachable floor, and a judge that finally distinguishes coincidence from interference. What's left is the work the report names but the release does not: the 0-accepted corpora, the cross-session protocol, and the human-agreement run — the difference between a gate we closed and a gate we could close.
CauterRule v0.3.1 is released. The scorer before/after, the safety CIs, and the J1/J12/J16 issue journal are in the field test report. The repo is public. Install with
pip install cauterule. Changelog · Release notes