Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.
pip install cauterulegives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.1 field test report evaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the source for every number below. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
The last article ended on an open question: at what point does a rule that is right-but-reworded deserve credit, and who decides? For v0.3.1 we built the instrument that question deserves — a direct extraction-accuracy metric, scoring the extracted rule against the expected_rule ground truth the corpus already carried but we had never used.
The first reading was 0.08.
We stared at it the way we stared at every low number in this project's history: reach for the model. Then we looked at the ruler. The ruler was broken — in the same way we had accused the replay gate of being broken, which makes this article feel a little like we are grading our own homework on a rubric we just wrote. It is also the most satisfying fix of the release cycle.
Four of the 40 corpora carry an expected_rule field — the ground-truth rule a trajectory should produce: golden (60, backfilled with the authored scenarios in #735), failures/positive (23 trajectories carry one), reference-expansion (303), and its paraphrase-diversity slice (15). For every extracted candidate we now compute, per corpus:
| Metric | Comparator | What it grades |
|---|---|---|
token_f1 |
literal token F1, extracted vs expected_rule
|
surface form |
semantic_f1 |
MiniLM cosine on the full rule | meaning, whole rule |
directive_f1 |
literal token F1 on the do half only |
the directive phrase |
agreement |
trigger-only semantic match (after J6) | the headline |
The metric's job is narrow and it matters: it answers "did the model extract the right rule?" independently of replay, which answers "can we verify it against history?" One number, two questions — the split from the previous article, finally instrumented.
The first version gated agreement on the directive's literal token F1. A directive like "pull latest changes before pushing" is a short phrase. A correct rewording — "run a rebase or fetch-merge first" — shares almost no tokens with it and scores ~0.22. Gate a composite on that, and the headline agreement on golden collapses to 0.08–0.10.
Read that number naively and the story is "the model is bad at extracting directives." Read the four sub-metrics side by side and the story is different: the trigger — the part of the rule that identifies the failure — was matching at semantic similarity of 0.6+. The model knew when. The gate was punishing it for rephrasing what to do.
That is the replay sin from the last article, committed by the new metric: a lexical comparator wearing a validation costume. We had diagnosed the disease in one half of the pipeline and then prescribed the same drug for the other half.
Two changes:
agreement is now the trigger-only semantic match. The directive stops gating the headline; it is reported alongside as directive_f1 so nobody loses the signal — it just stops deciding it.The regression test that pins this is test_agreement_is_trigger_only_directive_not_gated in tests/measurement/test_extraction_accuracy.py. A metric, once it gates a release, is load-bearing code. It gets tests like anything else.
J10 — 0.0 is a value; null is a different value. The first version of the metric emitted extraction_f1: 0.0 for corpora with no expected_rule at all (raw/ci). But 0.0 means measured, and the answer was zero — which on a corpus with no ground truth is a lie. It is indistinguishable from total failure. to_dict() now returns null when n == 0, and the runner emits None. The report can now tell no ground truth from ground truth said zero.
J13 — label the rate for what the corpus actually is. raw/opencode was reporting a "false_accept_rate" of 0.68 / 0.72 across the two models. It is a promotion corpus, not a rejection corpus — that number is the acceptance rate, i.e. working as intended. Calling it "false accept" makes a pass rate read like a safety violation in every summary file we shipped. Rejection corpora (nearmiss, adversarial/*) report false_accept_rate; silence and extraction corpora report acceptance_rate.
Neither fix changes a model or a threshold. Both change whether the report means what it says.
Cloud models, v0.3.1, from the committed artifacts:
| Corpus (n) | Model | token_f1 | semantic_f1 | directive_f1 | token_agree | agreement |
|---|---|---|---|---|---|---|
| golden (60) | gpt-4o-mini | 0.419 | 0.611 | 0.205 | 0.250 | 0.783 |
| golden (60) | llama-3.1-8b | 0.484 | 0.670 | 0.220 | 0.367 | 0.850 |
| failures/positive (23) | gpt-4o-mini | 0.527 | 0.664 | 0.302 | 0.304 | 0.739 |
| failures/positive (23) | llama-3.1-8b | 0.556 | 0.652 | 0.294 | 0.478 | 0.783 |
| reference-expansion (303) | gpt-4o-mini | 0.608 | 0.734 | 0.244 | 0.548 | 0.917 |
| reference-expansion (303) | llama-3.1-8b | 0.654 | 0.749 | 0.205 | 0.630 | 0.917 |
| paraphrase-diversity (15) | gpt-4o-mini | 0.512 | 0.718 | 0.311 | 0.267 | 0.933 |
| paraphrase-diversity (15) | llama-3.1-8b | 0.509 | 0.711 | 0.388 | 0.200 | 0.933 |
Look at the spread, not the single column:
Same model, same corpus, three comparators, three very different stories. The model extracts the right rule in different words. That was the v0.3.0 open question (plan Q2) — "is the extraction quality gap the model, or the matcher proxy?" — and this is the empirical answer: the proxy.
expected_rule needed no new labeling — it was being parsed out and dropped at load. Picking it up is the cheapest high-signal metric we have, and we waited a full release to pick it up.null as a first-class value. Distinguishing unmeasurable from zero is a one-line change that prevents an entire class of misread reports.test_extraction_accuracy.py pins the trigger-only definition; J6's behavior is a test, not a comment.0.0 for "not measured." It read as total failure in summary.json and would have polluted any downstream average silently.false_accept_rate on a promotion corpus is a rate with the wrong sign of connotation. Labels are interpretation; interpret the corpus, not the habit.agreement says the extraction was good. Is a right-when/wrong-do rule "extracted correctly"? We report directive_f1 but do not act on it. That is a decision we have deferred, not solved.failures/positive has 23 ground-truth trajectories and paraphrase-diversity 15. Agreement at n=15 is a direction, not a claim. The Wilson CIs we now publish for pass rates do not yet cover these extraction numbers.reference-expansion's 0.917 overfit to our authored references? The 303 trajectories are paraphrase-expansions of our own reference pool. High agreement there partly means "the model reproduces our phrasings." How much of 0.917 transfers to rules nobody in the pool wrote? We do not yet know.A metric is untested code until it is tested. We wrote a measurement instrument with the same confidence we would write a load-bearing feature — and it shipped a lying first version. The fix was structural (trigger-only, parts reported) but the lesson is procedural: the ruler gets tests before the reading gets believed.
null is a value, and so is a label. 0.0 vs null and false_accept_rate vs acceptance_rate are not cosmetics. A report that cannot distinguish unmeasured from zero, or working as intended from safety violation, is a report that will be misread by the next person — usually under time pressure, which is when reports are read.
The spread between comparators is the diagnosis. token_f1 0.5, semantic 0.7, directive 0.2 — read together, that triple says exactly where a rewording-resistant rule lives. A single composite says "medium." Composites compress; sub-metrics localize.
When a new metric contradicts the old narrative, suspect the metric first — including the new one. 0.08 implied the model was bad. 0.78 implied the model was good. Both readings were one comparator choice away from being wrong. The data did not change between the two; the ruler did.
If you score a generated artifact against ground truth, you are not adding a number — you are adding a second system with its own bugs, sitting between you and the truth. Test it against cases where you know the answer (a near-verbatim extraction must score high; a reworded-but-correct one must not collapse), publish its parts, and give "unmeasurable" its own value. Otherwise you will spend a cycle fixing the thing the metric was designed to exonerate.
The honest ruler pointed at the next problem: two corpora had sat at a flat zero for a full release, and the reason was not in the model at all.
expected_rule)CauterRule v0.3.1 is released. The extraction-accuracy tables, the J6/J10/J13 issue-journal entries, and the full 40-corpus × 2-model sweep are in the field test report. The repo is public. Install with
pip install cauterule. Changelog · Release notes