Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.
pip install cauterulegives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.1 field test report evaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the source for every number below. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
Every release, the field test produces a list of worst corpora, and every release the honest first guess is the same: the model can't do this kind of failure. The two worst carried into the v0.3.1 cycle from the v0.3.0 report: adapters — 0/60 on both models, 100% matcher_gap — and raw/ci — 0/110.
Both diagnoses were wrong. Both fixes were data. And both corpora now pass: 60/60 and 21–26/47.
That is the whole article: a 0% corpus is a question, not a verdict, and the question you should ask before touching the model is "what is in the haystack?"
matcher_gap is not no_signal
Replay attributes every inconclusive to a reason. The two that matter here are different animals:
| Attribution | What it means | Who is at fault |
|---|---|---|
no_signal |
the model produced no candidate, or the candidate matched nothing and there was nothing to match | model (or gate) |
matcher_gap |
the model produced a candidate, and the reference pool returned nothing it could be compared to | the pool |
adapters was 100% matcher_gap. The model was happily extracting LangGraph, CrewAI, and PydanticAI rules. They just had nowhere to land: the reference pool had zero framework-specific trajectories. A candidate with nothing to match against is not evidence the candidate is bad. It is evidence the haystack is empty in that domain.
raw/ci told a messier version of the same story — and the haystack was broken in four layers at once.
The reference pool for v0.3.0 had no langgraph/crewai/pydanticai trajectories at all, so every adapter candidate fell back to a pool with no same-domain references and scored matcher_gap. The fix was not a matcher change. It was authoring adapter-specific reference signatures with matching domains — same shape, same domain label, real failure text.
The result, both models, no model change:
| Corpus | v0.3.0 | v0.3.1 | |
|---|---|---|---|
adapters (n=60) |
0 pass (100% matcher_gap) |
60/60 pass | precision 1.000, recall 0.168 (gpt) / 0.175 (llama) |
0 → 100% in one cycle. The strongest before/after in the sweep. The model had been right all along; the pool had been empty.
raw/ci went from 0/110 to 21/47 (gpt) and 26/47 (llama) — a ~45–55% pass corpus out of a zero corpus. Four separate data bugs, each one independently capable of producing a zero:
collect-ci-corpus-v2.py's predecessor stored the first 2000 characters of gh run view --log — the CI runner's boot/checkout boilerplate. The actual failure is at the bottom of a CI log. So 99 of 110 trajectories had no failure text in them. The model was being asked to extract a rule from a log that did not contain the failure. Re-fetch the tail, re-derive failure_class.generate_ci_replay_refs.py); the pool grew 540 → 588.failure_class label was mixed into the embedded signature, dragging short paraphrase triggers under the floor (more on the floor itself in the next article). The matcher now takes the max similarity over a class-free failure view.Layer 1 is the one that generalizes furthest. A log collector that grabs the head of a file is measuring boilerplate and calling it the event. If your "failure text" field is ever empty more often than you expect, check which end of the stream you are slicing before you check the model.
Once the haystack was real, the two models diverged in an instructive way on raw/ci — 21/47 for gpt-4o-mini, 26/47 for llama-3.1-8b, with 11/47 trajectories discordant (8 llama-only passes vs 3 gpt-only). The paired-delta analysis (J15) showed why: gpt writes repo-specific triggers — exact error codes, module names — that score 0.0–0.11 against generic sibling references, while llama's broader phrasings match better.
We did not fix that. We accepted it (J14) and reported it, because the honest root cause is a model/style gap against a reference pool that is still too generic to reward specificity. One confession belongs in this article too: the earlier plan to "lower the raw/ci threshold from 0.45 to 0.40" was based on a dead code path — CORPUS_THRESHOLDS is never read; the live threshold for raw/* is 0.35 via threshold_for_corpus(). A threshold change that moves no number is a threshold you have not traced to the call site.
matcher_gap as a first-class signal. It is the pool's name on the problem. A corpus that is 100% matcher_gap should trigger "inventory the reference pool," not "upgrade the model."adapters and raw/ci, the fix was adding references the candidate was actually about. The pool is a product surface, not a fixture.CORPUS_THRESHOLDS to zero call sites.public/staleness 0/10, public/synthetic 0/10, lifecycle 0/40, mcp 0/20, gpt public/domains 0/30. For each: coverage gap (fix with references) or matcher gap (fix with config) or model gap (accept)? diagnose_corpus.py exists to answer this per corpus; the answers are the v0.4.0 work list.adapters went 0→60/60 with a modest reference addition. golden needed 54 authored replay references. Is there a knee in the curve, or do thin domains (raw/sibling-repos: 1/10 on both models) keep starving?public/* corpora are the honest check on whether authored-pool gains transfer, and they are precisely the ones still at zero.A 0% corpus is a question. "What can the grader see?" — and the answer is usually the reference pool, the corpus collector, or both. The model is the last suspect, not the first.
matcher_gap is a pointer to the data layer. When a candidate exists and nothing in the pool resembles it, the pool is on the hook. Read the attribution before you read the rate.
The reference pool is a product surface. It decides what a rule can be compared against, which means it decides what counts as verified. Treating it as a static fixture is how you ship a ruler with holes.
Check which end of the stream you sliced. A log, a trace, a transcript — the event is rarely at the top. Collectors that grab the head produce "weak model" readings that are really "empty input" readings.
Before you blame the model, inventory what the grader can see. If the answer isn't in the haystack, no model scores — and the 0% you are reading is a property of the evaluation, not the capability.
The haystack is full now. The next problem was a number that sat between the model and the verdict: a similarity floor set at 0.80, in a world where honest paraphrase matches land at 0.63.
CauterRule v0.3.1 is released. The adapter and CI before/after, the paired model deltas, and the J11/J18 issue journal are in the field test report. The repo is public. Install with
pip install cauterule. Changelog · Release notes