The experiment was fine. The runner was the bug.
A single hung API call discarded hours of completed work, and I kept blaming the data.
If your long-running eval keeps dying at 80%, this is the five-part fix I wish I'd written first.
I run field tests for
CauterRule
, an OSS sidecar that learns standing rules from repeated agent failures. I had spent weeks on extraction, matching, and scoring — and almost no time on the process that runs everything.
A Partial Sweep Still Looks Like Data
The first time the benchmark died, I assumed the corpus was the problem. It died again. Same place.
One hung API call had discarded 60 completed audits.
No partial report, no error record, just a run that stopped and a directory that looked emptier than it should.
That's the trap: a partially completed sweep doesn't announce itself. It produces a file of results that
looks
like data. If you report on it, you've quietly changed your denominator, and you won't know which numbers are real.
A hang is not a performance issue. It's a
data-integrity
issue.
The Five-Part Fix
Failure mode
Before
After
Hung API call
run dies at 80%, 60 audits lost
per-trajectory timeout + non-retryable classification + quarantine
Runaway response
uncapped tokens
token cap per trajectory
Crash mid-sweep
partial report, no record
cancel-on-shutdown + quarantine record
Cost tracking
not captured
token capture + per-model pricing
The mechanisms, in plain terms:
Per-trajectory timeout
— no single call can hang the sweep.
Non-retryable timeout classification
— a timeout is recorded as
timeout
, not retried forever or mistaken for a model failure.
Token cap
— a runaway response can't silently inflate a run.
Quarantine on failure
— a failed trajectory is set aside with its error, not dropped.
Cancel-on-shutdown
— an interrupted sweep writes what it has and marks the rest, instead of vanishing.
Tracked in issue #713. The result was a clean field test of
4,768 trajectory-runs
across 40 corpora and 2 cloud models, with
zero lost sweeps
— see the
v0.3.0 field test report
. Per-model cost capture is in the
cost measurement doc
.
Why This Is the Unglamorous Part
Resilience work feels like a distraction from the "real" ML problem. But every long run that dies halfway is silently poisoning the results, because a partially completed sweep still looks like data — and you won't know which run was real.
The fix wasn't a better model or a bigger machine. It was treating a single hung call as a
first-class failure state
instead of an accident to re-run.
Lessons
A hang is a data-integrity bug, not a performance bug.
Fail closed and record it.
A quarantined failure is worth more than a silent gap.
Cap the blast radius.
Per-trajectory limits beat per-sweep heroics.
Track cost per model from the start
— you can't reason about routing without it.
The Honest Limitation
Runner hardening adds latency and complexity, and a timeout can mask a real model problem by treating it as infrastructure noise. I chose to fail closed and record every timeout, but the boundary between "transient" and "broken" is judgment, not science.
Not every hang is infrastructure. Some are the model genuinely struggling — and hardening the runner means you must still read the quarantine to tell the difference.
What's the ugliest piece of infrastructure your eval depends on? Mine was a runner I didn't respect for three weeks.
Code and receipts:
CauterRule
·
v0.3.0 field test report
·
cost measurement
·
CHANGELOG
— all MIT, all public.