Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.
pip install cauterulegives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.0 field test report evaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
For two field tests, one number refused to move: golden recall sat at 0.087. The reports both named the same culprit — "replay and matcher calibration is now the highest-value engineering target." So we did the obvious thing and worked on the matcher.
The matcher was not the problem. The denominator was.
Replay grades a candidate trigger against reference trajectories. Recall is the fraction of failures the trigger catches:
precision = prevented / (prevented + broken)
recall = prevented / total_failures
A v0.2.0 candidate that prevented 3 real failures — a genuinely useful rule — was scored 3 / ~200 = 0.015. No threshold can admit 0.015 without also admitting noise. The rule wasn't weak. The denominator was the entire corpus.
Ask the question the metric was actually answering: "does this git rule also prevent docker, python, terraform, and browser failures?" Of course it doesn't. It was never supposed to.
We built replay on one implicit assumption:
Assumption: every failure in the reference pool is a fair test case for every candidate rule.
That assumption is wrong whenever candidates are domain-specific — which, for a real rule engine, is always. The consequence is subtle and dangerous: the metric doesn't fail loudly. It just makes good rules look worthless, uniformly, across every model. And a metric that makes everything look equally bad is a metric that hides where the actual problem is.
We also assumed recall was a model measurement. It is a ratio, and we were controlling the denominator as if it were irrelevant.
v0.3.0 filters reference_trajs to the source trajectory's domain before replay:
# scope the comparison set to the candidate's own domain
references = [t for t in reference_pool if t.domain == candidate.source_domain]
prevented = sum(1 for t in references if rule_matches(candidate, t) and not t.success)
recall = prevented / len([t for t in references if not t.success])
The pool did not shrink — it grew from 230 to 444 references. What changed is the slice each candidate is judged against:
| Candidate domain | Scoped references | Global pool (old) |
|---|---|---|
| git | 19 | ~200 |
| python | 30 | ~200 |
| docker | 30 | ~200 |
A rule that prevents 3 git failures is now scored 3 / 19 ≈ 0.16, not 3 / 200 = 0.015.
We made one more change alongside it, and it deserves to be stated plainly: the pass threshold dropped from 0.8 to 0.5. That is not a matcher fix. It is the acknowledgement that once scores are computed honestly, 0.8 was calibrated to an inflated scale. Report both or you are hiding the trick.
Golden and failures/positive recall, both cloud models, pre- and post-fix:
| Corpus | Model | Recall pre | Recall post | Delta |
|---|---|---|---|---|
| golden | gpt-4o-mini | 0.068 | 0.170 | 2.5× |
| golden | llama-3.1-8b | 0.104 | 0.228 | 2.2× |
| failures/positive | gpt-4o-mini | 0.087¹ | 0.182 | ~2.1× |
| failures/positive | llama-3.1-8b | 0.087¹ | 0.259 | ~3.0× |
¹ Golden pre/post are the v0.3.0 pre/post domain-scoping runs (report §8). The report does not tabulate a separate failures/positive pre-run, so that pre column uses the v0.2.0 baseline (0.087); the post values are recomputed from the committed v0.3.0 artifacts (mean best-candidate recall).
No model change. No prompt change. No change to the core matcher scoring. Golden pass rate moved to 40% (gpt-4o-mini) and 50% (llama-3.1-8b) after the near-miss tolerance band. Verified caveat: the committed per-candidate verdicts show **30% for both* models (3/10); the 40–50% is the tolerance-band-applied value and the band-applied re-run was not committed to field-test/results/0.3.0 — see the field-test report's data-verification note.*
Evaluation-set composition is a product decision, not a detail. "Recall against all failures" sounds rigorous and is actually a different question than the one you care about.
Uniformly bad numbers across every model usually mean an evaluation bug. When gpt-4o-mini and llama-3.1-8b agree that everything is hopeless, suspect the denominator before the model.
Scoping is not cherry-picking — if it is principled and pre-registered. We scope by a trajectory property fixed before scoring. We are not removing failures a candidate missed; we are removing failures it was never meant to catch. If that boundary moves after you see the results, you are just tuning the metric until it flatters you.
Recall is a floor. Every reference you add lowers recall without changing the rule. Report it as "at least this many," never as "this many."
If you score a specific output against a set of examples, ask what the denominator means before you optimize the scorer.
CauterRule spent two releases blaming a matcher for a number the reference pool was manufacturing. The fix was not a better algorithm. It was recognizing that "recall" should mean recall within the problem this rule is about. Change the comparison slice and the same model — the same everything — looks 2–3× better and, more importantly, becomes diagnosable.
CauterRule v0.3.0 is released. The full recall data and the 40-corpus × 2-model matrix are in the field test report. The repo is public. Install with
pip install cauterule. Changelog · Release notes