{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "4,768 LLM Runs, Zero Lost Sweeps: Hardening a Field-Test Runner for Timeouts, Hangs, and Cost",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/4768-llm-runs-zero-lost-sweeps-hardening-a-field-test-runner-for-timeouts-hangs-and-cost-1k24",
  "publish_date": "2026-09-12",
  "corpus_stage": "",
  "notes": "",
  "id": "0f8b1f4d479f",
  "collected_at": "2026-10-02T14:06:12",
  "title": "4,768 LLM Runs, Zero Lost Sweeps: Hardening a Field-Test Runner for Timeouts, Hangs, and Cost - DEV Community",
  "body_markdown": "Update — v0.3.0 released.\nCauterRule is now live on\nGitHub\nand\nPyPI\n. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.\npip install cauterule\ngives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The\nv0.3.0 field test report\nevaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below.\nRelease notes\n·\nChangelog\nCauterRule\nis an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.\nHere is the failure mode that never makes it into a paper. You launch a 4,768-run benchmark. It churns for an hour. Then one request to the model provider simply never returns — no exception, no error, no timeout. The worker is blocked. The queue drains to zero throughput. You have not lost one trajectory. You have lost the sweep, and the API spend it already consumed.\nv0.3.0 is the first CauterRule field test that finished. The reason is not a better prompt. It is five small guards in the runner.\nThe assumptions we started with (all of them wrong at scale)\nOur original runner encoded three assumptions that hold at 50 runs and fail at 4,768:\nAssumption 1:\nthe provider will always respond.\nAssumption 2:\na timeout is a transient error, so retry it.\nAssumption 3:\na failed trajectory should be retried until it succeeds.\nAt scale, assumption 1 produces infinite hangs, assumption 2 doubles the cost of a request that was already dead, and assumption 3 turns one bad input into an unbounded retry loop. None of these are model problems. They are harness problems — and they compound.\nThe scale that breaks them:\n40 corpora × 2 models = 4,768 runs\n, roughly\n2,384 trajectories per model\n, several LLM calls per trajectory.\nThe fix: five guards (#713)\nNone of this is clever. It is the set of things you must do before you can trust a large sweep:\nGuard\nImplementation\nFailure it removes\nPer-trajectory timeout\nfuture.result(timeout=120)\n(client\ncreate(timeout=30)\n)\nA hung call blocks a worker forever\nNon-retryable timeouts\nclassify timeout as terminal\nRetrying a dead request multiplies cost\nToken cap\nmax_tokens=4096\nRunaway generations blow latency and spend\nQuarantine\nCAUTERULE_QUARANTINE_IDS\nKnown-bad IDs are skipped, not retried\nCancel on shutdown\nexecutor.shutdown(wait=False, cancel_futures=True)\nA stuck thread blocks\nThreadPoolExecutor.__exit__\nThe shape of the concurrency fix — note it is the per-future\nresult(timeout)\n, not a\nwith\nblock, that does the work. The\nwith ThreadPoolExecutor\ncontext manager's\n__exit__\ncalls\nshutdown(wait=True)\n, which blocks until every thread finishes, so a single stuck thread would hang the whole sweep:\nfutures\n=\n{\nexecutor\n.\nsubmit\n(\nrun_trajectory\n,\nt\n):\nt\nfor\nt\nin\ntrajectories\n}\nfor\nfut\nin\nas_completed\n(\nfutures\n):\ntry\n:\ncollect\n(\nfut\n.\nresult\n(\ntimeout\n=\n120\n))\n# per-trajectory watchdog\nexcept\nTimeoutError\n:\nmark\n(\nfutures\n[\nfut\n],\n\"\ntimeout\n\"\n)\n# record it, keep sweeping\nexecutor\n.\nshutdown\n(\nwait\n=\nFalse\n,\ncancel_futures\n=\nTrue\n)\n# never wait on a stuck thread\nEnter fullscreen mode\nExit fullscreen mode\nThe principle:\none bad request costs one trajectory, not the sweep.\nThe data\nWhat the hardening bought, concretely:\nMetric\nValue\nTrajectory-runs completed\n4,768\nModels\n2 (gpt-4o-mini, llama-3.1-8b)\nCorpora\n40\nSafety gate dropped (per model)\n580 trajectories\nLLM calls avoided by the gate (per model)\n1,160\nSafety silence (successes / negatives)\n60/60 · 60/60\nAdversarial promotions\n0\nThe safety number is only credible because every corpus finished. A benchmark that completes its easy corpora and dies on the hard ones is measuring the corpora it could finish.\nBonus: cost became a first-class metric\nOnce runs stop dying, you can afford to measure spend. v0.3.0 captures token usage on every call and prices it with real provider rates:\n@dataclass\nclass\nLLMResponse\n:\ntext\n:\nstr\nprompt_tokens\n:\nint\n=\n0\ncompletion_tokens\n:\nint\n=\n0\nEnter fullscreen mode\nExit fullscreen mode\nModel\nInput ($/1M)\nOutput ($/1M)\ngpt-4o-mini\n0.15\n0.60\nllama-3.1-8b\n0.06\n0.06\nThe SDK was already returning\nusage\non every response. We were discarding it. Adding two fields turns \"what does 1,000 trajectories cost?\" from a shrug into a measured answer — the question that actually gates adoption.\nWhat worked\nThe sweep completed end to end.\n40 corpora, 2 models, 4,768 runs.\nThe motivating hang is now a non-event.\nThe original\nci-fail-015\nprompt drove a local model into infinite repetitive generation and blocked the sweep indefinitely; with the per-future watchdog it completes in ~3.7s and\nraw/ci\nruns 110/110.\nFail-soft held.\nTimeouts and quarantine dropped broken\nruns\n, not hard\ncases\n— the results stayed comparable.\nCost transparency fell out for free.\nA few lines of token accounting, using data already on the wire.\nA single hung call is now a rounding error.\nThe failure mode that motivated the work is gone.\nWhat didn't work\nLocal models did not survive contact with scale.\nThe same hardening made it undeniable that the local OMLX setup was too slow and hung on\nraw/ci\n. The field test went cloud-only. The \"ship on local models\" path from the v0.1.0 report did not hold at 40 corpora.\nWe built measurement tools and didn't run the protocols.\nCross-session repeat-failure reduction and human-vs-replay agreement both have scripts (\ncross_session.py\n,\nhuman_agreement.py\n) and runner flags. Neither protocol was executed. A flag that was never pulled is not evidence.\nWe didn't instrument the drops.\nThe runner discards timed-out trajectories cleanly, but we did not log the timeout distribution. The shape of the failure — where and how often calls hang — is itself data we threw away.\nQuestions we still can't answer\nIs 120s the right timeout, or does it truncate legitimately slow multi-step extractions? We never measured how close normal calls come to the ceiling.\nDo the timeouts cluster in\nraw/ci\n, or are they spread evenly? We can't say without per-drop logging.\nDoes quarantine mask a provider regression? Skipping known-bad IDs is safe for a one-off sweep; in a continuous loop it could hide a systematic failure.\nWhat is the actual $/1k-trajectory number per corpus, not just per model? We have per-model pricing but haven't published per-corpus costs.\nWhat I learned\nThroughput is a product feature for evaluation.\nYour benchmark's quality ceiling is whether it finishes. A harness that can't survive a hung call can't run the corpora where the interesting failures live.\nFail-soft beats fail-hard.\nRetries that assume transience are a cost bug waiting to happen. Classify failures as terminal vs. transient explicitly.\nCapture usage while you're already holding the client.\nToken accounting is trivial when you own the call site and impossible when you don't. Do it from day one.\nBuilding the instrument is not taking the reading.\nWe shipped cross-session and agreement tooling and left both protocols unrun, so those questions are still open. Tooling is not measurement.\nThe broader lesson\nIf you run large evaluations against a flaky external API, the infrastructure is not a distraction from the science — it\nis\npart of the science.\nThe most valuable code in CauterRule's v0.3.0 field test was not a matcher or a prompt. It was five guards that let 4,768 requests finish, plus two dataclass fields that let us price them. That is what turned a benchmark that kept dying into one you can trust — and, conveniently, one that can tell you what it costs.\nReferences\nCauterRule v0.3.0 release notes\nv0.3.0 field test report\nCost measurement (token capture + per-model pricing)\nField test plan (corpora, models, runner flags)\nGenerated results (all 40 corpora)\nPRD — architecture\nUser guide\n·\nChangelog\nCauterRule v0.3.0 is released.\nThe methodology, hardening notes, and cost report are in the\nfield test report\nand the\ncost measurement doc\n. The\nrepo\nis public. Install with\npip install cauterule\n.\nChangelog\n·\nRelease notes",
  "metadata": {
    "title": "4,768 LLM Runs, Zero Lost Sweeps: Hardening a Field-Test Runner for Timeouts, Hangs, and Cost - DEV Community",
    "description": "Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent... Tagged with ai, agents, healthydebate, llm.",
    "keywords": "ai, agents, healthydebate, llm, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/4768-llm-runs-zero-lost-sweeps-hardening-a-field-test-runner-for-timeouts-hangs-and-cost-1k24",
    "og:title": "4,768 LLM Runs, Zero Lost Sweeps: Hardening a Field-Test Runner for Timeouts, Hangs, and Cost",
    "og:description": "Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8syy5fb70y2v9z1hkl54.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "4,768 LLM Runs, Zero Lost Sweeps: Hardening a Field-Test Runner for Timeouts, Hangs, and Cost",
    "twitter:description": "Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8syy5fb70y2v9z1hkl54.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/0f8b1f4d479f/code/body-000.txt",
      "chars": 367,
      "language": "text"
    },
    {
      "file": "attachments/0f8b1f4d479f/code/body-001.txt",
      "chars": 102,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/deghosal-2026/CauterRule",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://pypi.org/project/cauterule/",
      "label": "PyPI",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md",
      "label": "v0.3.0 field test report",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/release/v0.3.0/release-notes.md",
      "label": "Release notes",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md",
      "label": "Changelog",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/cost-measurement.md",
      "label": "Cost measurement (token capture + per-model pricing)",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/field-test-plan.md",
      "label": "Field test plan (corpora, models, runner flags)",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/generated-results.md",
      "label": "Generated results (all 40 corpora)",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/design/prd/02-architecture.md",
      "label": "PRD — architecture",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/USER_GUIDE.md",
      "label": "User guide",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:16Z",
  "source_url": "https://dev.to/debashish_ghosal/4768-llm-runs-zero-lost-sweeps-hardening-a-field-test-runner-for-timeouts-hangs-and-cost-1k24",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 8076,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/0f8b1f4d479f/pages/source.html",
  "local_html": "attachments/0f8b1f4d479f/pages/article.html",
  "body_base_path": "attachments/0f8b1f4d479f/pages/",
  "body_markdown_path": "attachments/0f8b1f4d479f/pages/article.md"
}