{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "One Hung API Call Used to Kill My 1,000-Run Benchmark. Here's the Fix.",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/one-hung-api-call-used-to-kill-my-1000-run-benchmark-heres-the-fix-555",
  "publish_date": "2026-09-26",
  "corpus_stage": "",
  "notes": "",
  "id": "58504c1eed8b",
  "collected_at": "2026-10-02T14:05:52",
  "title": "One Hung API Call Used to Kill My 1,000-Run Benchmark. Here's the Fix. - DEV Community",
  "body_markdown": "The experiment was fine. The runner was the bug.\nA single hung API call discarded hours of completed work, and I kept blaming the data.\nIf your long-running eval keeps dying at 80%, this is the five-part fix I wish I'd written first.\nI run field tests for\nCauterRule\n, an OSS sidecar that learns standing rules from repeated agent failures. I had spent weeks on extraction, matching, and scoring — and almost no time on the process that runs everything.\nA Partial Sweep Still Looks Like Data\nThe first time the benchmark died, I assumed the corpus was the problem. It died again. Same place.\nOne hung API call had discarded 60 completed audits.\nNo partial report, no error record, just a run that stopped and a directory that looked emptier than it should.\nThat's the trap: a partially completed sweep doesn't announce itself. It produces a file of results that\nlooks\nlike data. If you report on it, you've quietly changed your denominator, and you won't know which numbers are real.\nA hang is not a performance issue. It's a\ndata-integrity\nissue.\nThe Five-Part Fix\nFailure mode\nBefore\nAfter\nHung API call\nrun dies at 80%, 60 audits lost\nper-trajectory timeout + non-retryable classification + quarantine\nRunaway response\nuncapped tokens\ntoken cap per trajectory\nCrash mid-sweep\npartial report, no record\ncancel-on-shutdown + quarantine record\nCost tracking\nnot captured\ntoken capture + per-model pricing\nThe mechanisms, in plain terms:\nPer-trajectory timeout\n— no single call can hang the sweep.\nNon-retryable timeout classification\n— a timeout is recorded as\ntimeout\n, not retried forever or mistaken for a model failure.\nToken cap\n— a runaway response can't silently inflate a run.\nQuarantine on failure\n— a failed trajectory is set aside with its error, not dropped.\nCancel-on-shutdown\n— an interrupted sweep writes what it has and marks the rest, instead of vanishing.\nTracked in issue #713. The result was a clean field test of\n4,768 trajectory-runs\nacross 40 corpora and 2 cloud models, with\nzero lost sweeps\n— see the\nv0.3.0 field test report\n. Per-model cost capture is in the\ncost measurement doc\n.\nWhy This Is the Unglamorous Part\nResilience work feels like a distraction from the \"real\" ML problem. But every long run that dies halfway is silently poisoning the results, because a partially completed sweep still looks like data — and you won't know which run was real.\nThe fix wasn't a better model or a bigger machine. It was treating a single hung call as a\nfirst-class failure state\ninstead of an accident to re-run.\nLessons\nA hang is a data-integrity bug, not a performance bug.\nFail closed and record it.\nA quarantined failure is worth more than a silent gap.\nCap the blast radius.\nPer-trajectory limits beat per-sweep heroics.\nTrack cost per model from the start\n— you can't reason about routing without it.\nThe Honest Limitation\nRunner hardening adds latency and complexity, and a timeout can mask a real model problem by treating it as infrastructure noise. I chose to fail closed and record every timeout, but the boundary between \"transient\" and \"broken\" is judgment, not science.\nNot every hang is infrastructure. Some are the model genuinely struggling — and hardening the runner means you must still read the quarantine to tell the difference.\nWhat's the ugliest piece of infrastructure your eval depends on? Mine was a runner I didn't respect for three weeks.\nCode and receipts:\nCauterRule\n·\nv0.3.0 field test report\n·\ncost measurement\n·\nCHANGELOG\n— all MIT, all public.",
  "metadata": {
    "title": "One Hung API Call Used to Kill My 1,000-Run Benchmark. Here's the Fix. - DEV Community",
    "description": "The experiment was fine. The runner was the bug.   A single hung API call discarded hours of... Tagged with ai, llm, programming, benchmark.",
    "keywords": "ai, llm, programming, benchmark, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/one-hung-api-call-used-to-kill-my-1000-run-benchmark-heres-the-fix-555",
    "og:title": "One Hung API Call Used to Kill My 1,000-Run Benchmark. Here's the Fix.",
    "og:description": "The experiment was fine. The runner was the bug.   A single hung API call discarded hours of...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1e4ofdaa45g6lir6vxfd.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "One Hung API Call Used to Kill My 1,000-Run Benchmark. Here's the Fix.",
    "twitter:description": "The experiment was fine. The runner was the bug.   A single hung API call discarded hours of...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1e4ofdaa45g6lir6vxfd.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/deghosal-2026/CauterRule",
      "label": "CauterRule",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md",
      "label": "v0.3.0 field test report",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/cost-measurement.md",
      "label": "cost measurement doc",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md",
      "label": "CHANGELOG",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:10Z",
  "source_url": "https://dev.to/debashish_ghosal/one-hung-api-call-used-to-kill-my-1000-run-benchmark-heres-the-fix-555",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 3497,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/58504c1eed8b/pages/source.html",
  "local_html": "attachments/58504c1eed8b/pages/article.html",
  "body_base_path": "attachments/58504c1eed8b/pages/",
  "body_markdown_path": "attachments/58504c1eed8b/pages/article.md"
}