{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "0/60 Wasn't the Model: The Empty Haystack Behind My Two Worst Corpora",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/060-wasnt-the-model-the-empty-haystack-behind-my-two-worst-corpora-34nh",
  "publish_date": "2026-09-14",
  "corpus_stage": "",
  "notes": "",
  "id": "fc8525c07dec",
  "collected_at": "2026-10-02T14:06:07",
  "title": "0/60 Wasn't the Model: The Empty Haystack Behind My Two Worst Corpora - DEV Community",
  "body_markdown": "Update — v0.3.1 released.\nCauterRule is now live on\nGitHub\nand\nPyPI\n. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.\npip install cauterule\ngives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The\nv0.3.1 field test report\nevaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the source for every number below.\nRelease notes\n·\nChangelog\nCauterRule\nis an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.\nEvery release, the field test produces a list of worst corpora, and every release the honest first guess is the same:\nthe model can't do this kind of failure.\nThe two worst carried into the v0.3.1 cycle from the v0.3.0 report:\nadapters\n—\n0/60\non both models, 100%\nmatcher_gap\n— and\nraw/ci\n—\n0/110\n.\nBoth diagnoses were wrong. Both fixes were data. And both corpora now pass: 60/60 and 21–26/47.\nThat is the whole article: a 0% corpus is a\nquestion\n, not a verdict, and the question you should ask before touching the model is \"what is in the haystack?\"\nThe diagnostic that changed the plan:\nmatcher_gap\nis not\nno_signal\nReplay attributes every inconclusive to a reason. The two that matter here are different animals:\nAttribution\nWhat it means\nWho is at fault\nno_signal\nthe model produced no candidate, or the candidate matched nothing and there was nothing to match\nmodel (or gate)\nmatcher_gap\nthe model\nproduced a candidate\n, and the reference pool returned nothing it could be compared to\nthe pool\nadapters\nwas 100%\nmatcher_gap\n. The model was happily extracting LangGraph, CrewAI, and PydanticAI rules. They just had nowhere to land: the reference pool had\nzero\nframework-specific trajectories. A candidate with nothing to match against is not evidence the candidate is bad. It is evidence the haystack is empty in that domain.\nraw/ci\ntold a messier version of the same story — and the haystack was broken in four layers at once.\nFix 1 (#726): the adapters pool was empty. Fill it.\nThe reference pool for v0.3.0 had no langgraph/crewai/pydanticai trajectories at all, so every adapter candidate fell back to a pool with no same-domain references and scored\nmatcher_gap\n. The fix was not a matcher change. It was authoring\nadapter-specific reference signatures with matching domains\n— same shape, same domain label, real failure text.\nThe result, both models, no model change:\nCorpus\nv0.3.0\nv0.3.1\nadapters\n(n=60)\n0 pass (100%\nmatcher_gap\n)\n60/60 pass\nprecision 1.000, recall 0.168 (gpt) / 0.175 (llama)\n0 → 100% in one cycle. The strongest before/after in the sweep. The model had been right all along; the pool had been empty.\nFix 2 (J11): the CI haystack was broken in four layers\nraw/ci\nwent from\n0/110 to 21/47 (gpt) and 26/47 (llama)\n— a ~45–55% pass corpus out of a zero corpus. Four separate data bugs, each one independently capable of producing a zero:\nThe collector read the wrong end of the log.\ncollect-ci-corpus-v2.py\n's predecessor stored the\nfirst\n2000 characters of\ngh run view --log\n— the CI runner's boot/checkout boilerplate. The actual failure is at the\nbottom\nof a CI log. So\n99 of 110 trajectories had no failure text in them.\nThe model was being asked to extract a rule from a log that did not contain the failure. Re-fetch the tail, re-derive\nfailure_class\n.\nThe corpus was padded with noise.\n34 infra-only + 28 bogus trajectories carried no signal. Deleted: 110 → 48.\nThe reference bucket was generic.\n18 generic CI references for a corpus of real, repo-specific CI failures. Authored\n48 same-domain CI sibling references\n(\ngenerate_ci_replay_refs.py\n); the pool grew 540 → 588.\nThe signature was diluted.\nThe\nfailure_class\nlabel was mixed into the embedded signature, dragging short paraphrase triggers under the floor (more on the floor itself in the next article). The matcher now takes the max similarity over a class-free failure view.\nLayer 1 is the one that generalizes furthest. A log collector that grabs the head of a file is measuring boilerplate and calling it the event. If your \"failure text\" field is ever empty more often than you expect, check which end of the stream you are slicing before you check the model.\nThe model gap that remained (and was accepted)\nOnce the haystack was real, the two models diverged in an instructive way on\nraw/ci\n— 21/47 for gpt-4o-mini, 26/47 for llama-3.1-8b, with 11/47 trajectories discordant (8 llama-only passes vs 3 gpt-only). The paired-delta analysis (J15) showed why: gpt writes\nrepo-specific\ntriggers — exact error codes, module names — that score 0.0–0.11 against generic sibling references, while llama's broader phrasings match better.\nWe did not fix that. We\naccepted\nit (J14) and reported it, because the honest root cause is a model/style gap against a reference pool that is still too generic to reward specificity. One confession belongs in this article too: the earlier plan to \"lower the raw/ci threshold from 0.45 to 0.40\" was based on a dead code path —\nCORPUS_THRESHOLDS\nis never read; the live threshold for\nraw/*\nis 0.35 via\nthreshold_for_corpus()\n. A threshold change that moves no number is a threshold you have not traced to the call site.\nWhat worked\nTreating\nmatcher_gap\nas a first-class signal.\nIt is the pool's name on the problem. A corpus that is 100%\nmatcher_gap\nshould trigger \"inventory the reference pool,\" not \"upgrade the model.\"\nAuthoring same-domain references.\nFor both\nadapters\nand\nraw/ci\n, the fix was adding references the candidate was actually about. The pool is a product surface, not a fixture.\nRepairing the corpus before tuning the model.\n62 deleted trajectories, one collector rewrite, 48 new references — then, and only then, the pass rate became a measurement of the model instead of a measurement of the collector.\nPaired per-trajectory deltas\n(J15) made the gpt/llama split diagnosable instead of a \"one model is better\" hand-wave.\nWhat didn't work\nThe v0.3.0 read of 0/60 as a capability verdict.\n\"The model can't extract framework rules\" was written in a report before anyone checked whether the pool contained a single framework trajectory. A zero on an empty haystack is the emptiest of signals.\nThe head-of-log collector.\nIt ran for a release and a half, feeding the model boilerplate, and the failure was invisible because the model's \"weakness\" was exactly where boilerplate produces no rule.\nThreshold reasoning on dead code.\nThe 0.45→0.40 option cost real thinking time before someone traced\nCORPUS_THRESHOLDS\nto zero call sites.\nQuestions we still can't answer\nJ18 — the 0-accepted corpora that survived.\npublic/staleness\n0/10,\npublic/synthetic\n0/10,\nlifecycle\n0/40,\nmcp\n0/20, gpt\npublic/domains\n0/30. For each: coverage gap (fix with references) or matcher gap (fix with config) or model gap (accept)?\ndiagnose_corpus.py\nexists to answer this per corpus; the answers are the v0.4.0 work list.\nHow many references per domain before returns flatten?\nadapters\nwent 0→60/60 with a modest reference addition.\ngolden\nneeded 54 authored replay references. Is there a knee in the curve, or do thin domains (\nraw/sibling-repos\n: 1/10 on both models) keep starving?\nDo authored references overfit the test?\nWe author references\nin the domain of the corpus we then test on\n. That is how you fix coverage — but it is also how a self-graded loop quietly forms. The\npublic/*\ncorpora are the honest check on whether authored-pool gains transfer, and they are precisely the ones still at zero.\nWhat I learned\nA 0% corpus is a question.\n\"What can the grader see?\" — and the answer is usually the reference pool, the corpus collector, or both. The model is the last suspect, not the first.\nmatcher_gap\nis a pointer to the data layer.\nWhen a candidate exists and nothing in the pool resembles it, the pool is on the hook. Read the attribution before you read the rate.\nThe reference pool is a product surface.\nIt decides what a rule can be compared against, which means it decides what counts as verified. Treating it as a static fixture is how you ship a ruler with holes.\nCheck which end of the stream you sliced.\nA log, a trace, a transcript — the event is rarely at the top. Collectors that grab the head produce \"weak model\" readings that are really \"empty input\" readings.\nThe broader lesson\nBefore you blame the model, inventory what the grader can see. If the answer isn't in the haystack, no model scores — and the 0% you are reading is a property of the evaluation, not the capability.\nThe haystack is full now. The next problem was a number that sat between the model and the verdict: a similarity floor set at 0.80, in a world where honest paraphrase matches land at 0.63.\nReferences\nCauterRule v0.3.1 release notes\nv0.3.1 field test report (§6 per-corpus results, Appendix A J11/J14/J15/J18)\nCI corpus collector (tail-of-log, re-derived failure_class)\nCI sibling reference generator\n·\ngolden replay reference generator\nThreshold calibration (live vs dead threshold paths)\nAuto-generated per-corpus results\n·\nChangelog\nCauterRule v0.3.1 is released.\nThe adapter and CI before/after, the paired model deltas, and the J11/J18 issue journal are in the\nfield test report\n. The\nrepo\nis public. Install with\npip install cauterule\n.\nChangelog\n·\nRelease notes",
  "metadata": {
    "title": "0/60 Wasn't the Model: The Empty Haystack Behind My Two Worst Corpora - DEV Community",
    "description": "Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent... Tagged with ai, llm, agents, testing.",
    "keywords": "ai, llm, agents, testing, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/060-wasnt-the-model-the-empty-haystack-behind-my-two-worst-corpora-34nh",
    "og:title": "0/60 Wasn't the Model: The Empty Haystack Behind My Two Worst Corpora",
    "og:description": "Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx8362ukvzhtp2mkoyph2.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "0/60 Wasn't the Model: The Empty Haystack Behind My Two Worst Corpora",
    "twitter:description": "Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx8362ukvzhtp2mkoyph2.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/deghosal-2026/CauterRule",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://pypi.org/project/cauterule/",
      "label": "PyPI",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.1/FIELD_TEST_REPORT.md",
      "label": "v0.3.1 field test report",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/release/v0.3.1/release-notes.md",
      "label": "Release notes",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md",
      "label": "Changelog",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/scripts/collect-ci-corpus-v2.py",
      "label": "CI corpus collector (tail-of-log, re-derived failure_class)",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/scripts/generate_ci_replay_refs.py",
      "label": "CI sibling reference generator",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/scripts/generate_golden_replay_refs.py",
      "label": "golden replay reference generator",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.1/threshold-calibration.md",
      "label": "Threshold calibration (live vs dead threshold paths)",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.1/generated-results.md",
      "label": "Auto-generated per-corpus results",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:15Z",
  "source_url": "https://dev.to/debashish_ghosal/060-wasnt-the-model-the-empty-haystack-behind-my-two-worst-corpora-34nh",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 9367,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/fc8525c07dec/pages/source.html",
  "local_html": "attachments/fc8525c07dec/pages/article.html",
  "body_base_path": "attachments/fc8525c07dec/pages/",
  "body_markdown_path": "attachments/fc8525c07dec/pages/article.md"
}