Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.
pip install cauterulegives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.0 field test report evaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
For two releases we treated "the pass rate is low" as one problem and reached for the extractor. It turned out to be two problems we had fused into a single number. This is the split, and it changed where we point next.
The pipeline has two halves that answer two different questions:
Extraction: given a failure, does the model produce the right rule?
Replay / evaluation: given a rule, can we verify it against history?
We measured only the second, and we read its score as a verdict on the first. The replay gate that decides promotion computes:
precision = prevented / (prevented + broken)
recall = prevented / total_failures
where prevented and broken come from simulate(), which calls rule_matches() — a text matcher. So prevented means "the trigger's prose reached a similarity threshold against a reference failure's prose," and broken means "…against a reference success's prose." Nothing in that path asks whether applying the rule's directive would have changed the trajectory's outcome. The gate is a lexical resemblance check wearing a validation costume.
In the failures/positive corpus, trajectory F-001 is the canonical git case:
expected_rule (ground truth) |
"when git push fails with non-fast-forward, pull latest changes before pushing" |
extracted when
|
"when git push fails with non-fast-forward" |
extracted do
|
"pull latest changes before pushing" |
The model reproduced the reference rule almost verbatim. Extraction did its job. Then replay scored it:
failures_prevented = 5 successes_broken = 3 near_misses = 1
precision = 0.625 recall = 0.625 verdict = INCONCLUSIVE
Three "broken" successes were S-022-git-pull, S-023-git-status, and NM-044-git-commit-hook — a successful git status counted as broken by a git-push rule, because the two share the token git. A correct rule was demoted to inconclusive by three unrelated successes it merely rhymes with. The model wrote the right rule. The grader rejected it on the prose.
Assumption: the replay verdict tells us whether the extracted rule is good.
It tells us whether the extracted trigger's surface form resembles stored reference surface forms. Those are different claims. A correct rule phrased differently scores low; a wrong rule that happens to share vocabulary scores high. The metric is a text-similarity proxy, and it is the gate.
The telling part: the undercount is not unique to replay. The corpus carries a ground-truth rule (expected_rule) on 23/50 failures/positive trajectories and 288 reference-expansion trajectories — a ready-made extraction-accuracy instrument we were never scoring against. When we do score it naively (token-F1 of the extracted rule vs expected_rule), we get only ~0.58 (llama-3.1-8b) / ~0.50 (gpt-4o-mini). Not because the model is wrong — F-001 is near-verbatim — but because it rewords, and a token comparator can't see through rewording. The same paraphrase gap that breaks replay also undercounts extraction. We had one brittle text-matching lens in two places.
Cloud models, failures/positive (23 trajectories carry expected_rule):
| Metric | gpt-4o-mini | llama-3.1-8b | What it measures |
|---|---|---|---|
| Replay pass rate | 8% | 10% | trigger prose vs reference prose |
Extraction token-F1 vs expected_rule
|
0.50 | 0.58 | extracted rule vs ground truth (text) |
F-001 extracted vs expected_rule
|
near-verbatim | near-verbatim | a correct rule |
| F-001 replay verdict | inconclusive | inconclusive | the grader, not the rule |
The two left-column numbers look bad and point at the model. The right two columns say the model was fine and the measurements are the weak link. The spread is the whole story: extraction is substantively right but lexically variable, and every metric we ship is lexical.
expected_rule needs no new labeling on failures/positive — it was being parsed out and dropped. Picking it up is the cheapest high-signal metric we have.expected_rule is dropped at parse (Trajectory has no such field), so the only quality number was the replay verdict. We graded the homework with the wrong rubric for a full release.prevented from "text matched" into "outcome changed"?expected_rule dense enough to backfill onto golden (currently null) so extraction quality has a clean per-corpus number everywhere?A pass rate is a composite; decompose it before you optimize it. "8–10% pass" fused extraction and evaluation. The moment we split the number, the model exonerated itself and the grader took the hit.
Ground truth you don't score against is a decoration. The corpus already knew the answer (expected_rule). Not wiring it into a metric meant we optimized the wrong half while the right half went unmeasured.
If your "validation" only reads words, it can't validate meaning. The replay gate grades whether the trigger's prose rhymes with stored prose. That is a retrieval property, not a correctness property — and it is the gate.
One crisp failure case beats a dozen averages. F-001 is the entire diagnosis. When a metric makes a known-good rule fail, the metric is the bug, full stop.
If you build a loop that produces an artifact and then scores it, keep the two measurements separate and keep them honest. A score that grades surface form instead of behavior will make good outputs look bad and bad outputs that share vocabulary look fine — in both halves of the loop at once.
CauterRule spent two releases tuning an extractor that was already writing the right rule, because the number it was chasing was really a prose-similarity score. The fix is not a better model. It is (a) scoring extraction against the expected_rule we already have, with a comparator that can see past rewording, and (b) making the replay gate check whether the directive changes the outcome, not whether the trigger rhymes. Same model. Same corpus. Two honest metrics instead of one flattering lie.
expected_rule, domain labels)CauterRule v0.3.0 is released. The replay and extraction data are in the field test report. The repo is public. Install with
pip install cauterule. Changelog · Release notes