{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "Our Recall Was 0.087 and the Model Was Innocent: How Domain-Scoped Replay Doubled It",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/our-recall-was-0087-and-the-model-was-innocent-how-domain-scoped-replay-doubled-it-4ci4",
  "publish_date": "2026-09-12",
  "corpus_stage": "",
  "notes": "",
  "id": "362bf79f472f",
  "collected_at": "2026-10-02T14:06:13",
  "title": "Our Recall Was 0.087 and the Model Was Innocent: How Domain-Scoped Replay Doubled It - DEV Community",
  "body_markdown": "Update — v0.3.0 released.\nCauterRule is now live on\nGitHub\nand\nPyPI\n. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.\npip install cauterule\ngives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The\nv0.3.0 field test report\nevaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below.\nRelease notes\n·\nChangelog\nCauterRule\nis an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.\nFor two field tests, one number refused to move: golden recall sat at\n0.087\n. The reports both named the same culprit — \"replay and matcher calibration is now the highest-value engineering target.\" So we did the obvious thing and worked on the matcher.\nThe matcher was not the problem. The denominator was.\nThe symptom: a good rule scored almost zero\nReplay grades a candidate trigger against reference trajectories. Recall is the fraction of failures the trigger catches:\nprecision\n=\nprevented\n/\n(\nprevented\n+\nbroken\n)\nrecall\n=\nprevented\n/\ntotal_failures\nEnter fullscreen mode\nExit fullscreen mode\nA v0.2.0 candidate that prevented\n3 real failures\n— a genuinely useful rule — was scored\n3 / ~200 = 0.015\n. No threshold can admit 0.015 without also admitting noise. The rule wasn't weak. The denominator was the entire corpus.\nAsk the question the metric was actually answering:\n\"does this git rule also prevent docker, python, terraform, and browser failures?\"\nOf course it doesn't. It was never supposed to.\nThe assumption we made (and where it broke)\nWe built replay on one implicit assumption:\nAssumption:\nevery failure in the reference pool is a fair test case for every candidate rule.\nThat assumption is wrong whenever candidates are domain-specific — which, for a real rule engine, is always. The consequence is subtle and dangerous: the metric doesn't fail loudly. It just makes good rules look worthless, uniformly, across every model. And a metric that makes everything look equally bad is a metric that hides where the actual problem is.\nWe also assumed recall was a\nmodel\nmeasurement. It is a ratio, and we were controlling the denominator as if it were irrelevant.\nThe fix: scope the reference pool to the source domain (§708)\nv0.3.0 filters\nreference_trajs\nto the source trajectory's domain before replay:\n# scope the comparison set to the candidate's own domain\nreferences\n=\n[\nt\nfor\nt\nin\nreference_pool\nif\nt\n.\ndomain\n==\ncandidate\n.\nsource_domain\n]\nprevented\n=\nsum\n(\n1\nfor\nt\nin\nreferences\nif\nrule_matches\n(\ncandidate\n,\nt\n)\nand\nnot\nt\n.\nsuccess\n)\nrecall\n=\nprevented\n/\nlen\n([\nt\nfor\nt\nin\nreferences\nif\nnot\nt\n.\nsuccess\n])\nEnter fullscreen mode\nExit fullscreen mode\nThe pool did not shrink — it grew from 230 to\n444\nreferences. What changed is the slice each candidate is judged against:\nCandidate domain\nScoped references\nGlobal pool (old)\ngit\n19\n~200\npython\n30\n~200\ndocker\n30\n~200\nA rule that prevents 3 git failures is now scored\n3 / 19 ≈ 0.16\n, not\n3 / 200 = 0.015\n.\nWe made one more change alongside it, and it deserves to be stated plainly:\nthe pass threshold dropped from 0.8 to 0.5. That is not a matcher fix. It is the acknowledgement that once scores are computed honestly, 0.8 was calibrated to an inflated scale. Report both or you are hiding the trick.\nThe data\nGolden and failures/positive recall, both cloud models, pre- and post-fix:\nCorpus\nModel\nRecall pre\nRecall post\nDelta\ngolden\ngpt-4o-mini\n0.068\n0.170\n2.5×\ngolden\nllama-3.1-8b\n0.104\n0.228\n2.2×\nfailures/positive\ngpt-4o-mini\n0.087¹\n0.182\n~2.1×\nfailures/positive\nllama-3.1-8b\n0.087¹\n0.259\n~3.0×\n¹ Golden pre/post are the v0.3.0 pre/post domain-scoping runs (report §8). The report does not tabulate a separate failures/positive pre-run, so that pre column uses the v0.2.0 baseline (0.087); the post values are recomputed from the committed v0.3.0 artifacts (mean best-candidate recall).\nNo model change. No prompt change. No change to the core matcher scoring. Golden pass rate moved to\n40% (gpt-4o-mini)\nand\n50% (llama-3.1-8b)\nafter the near-miss tolerance band\n.\nVerified caveat: the committed per-candidate verdicts show **30% for both\n* models (3/10); the 40–50% is the tolerance-band-applied value and the band-applied re-run was not committed to\nfield-test/results/0.3.0\n— see the field-test report's data-verification note.*\nWhat worked\nRecall became diagnosable.\nAgainst a global pool, everything looked equally hopeless. Against a scoped pool, the recall signal separates a specific rule from a vague one.\nThe fix is model-independent.\nIt held across both models — the signature of an evaluation bug, not a capability ceiling.\nIt exposed the real bottleneck.\nOnce the denominator was honest, the remaining failures pointed at the matcher's inability to read paraphrases — a concrete, addressable problem.\nWhat didn't work\nRecall is still far below target.\n0.170–0.228 against a 0.70-ish ambition. Doubling a small number leaves a small number.\nThe paraphrase gap is untouched.\nSemantic matching runs at a\n0.2 blend weight\nand cannot bridge \"non-fast-forward\" against \"Updates were rejected because the remote contains work that you do not have locally.\" Same event, different tokens. The mechanism is worse than the weight suggests: the semantic term is capped at 0.2 of the blend and only floors the score to 0.70 at cosine ≥ 0.80, so a true paraphrase with\nzero\nshared tokens scores ~0.2 — below every threshold. Raising the weight alone would not fix it; the floor has to move, and the cosine has to be computed against the failure signature, not the whole trajectory.\nScoping could flatter reference-rich domains.\npython and docker have ~30 references; git has 19. We have not measured whether thin domains are systematically penalized.\nChanging two things at once blurs attribution.\nScoping and the threshold drop landed together; the 2–3× is scoping's, but the pass-rate movement is both.\nQuestions we still can't answer\nWould a 0.4–0.5 semantic weight close golden, or does the token-F1 term still dominate the blend?\nIs domain the right scope, or should it be failure-class? We chose domain because the label already exists on every trajectory — a convenience assumption.\nAt what reference count per domain does scoping stop helping? We have no curve, only two field tests.\nIf a candidate has no domain label, do we fall back to the global pool, and does that reintroduce the bug?\nWhat I learned\nEvaluation-set composition is a product decision, not a detail.\n\"Recall against all failures\" sounds rigorous and is actually a different question than the one you care about.\nUniformly bad numbers across every model usually mean an evaluation bug.\nWhen gpt-4o-mini and llama-3.1-8b agree that everything is hopeless, suspect the denominator before the model.\nScoping is not cherry-picking — if it is principled and pre-registered.\nWe scope by a trajectory property fixed before scoring. We are not removing failures a candidate missed; we are removing failures it was never meant to catch. If that boundary moves after you see the results, you are just tuning the metric until it flatters you.\nRecall is a floor.\nEvery reference you add lowers recall without changing the rule. Report it as \"at least this many,\" never as \"this many.\"\nThe broader lesson\nIf you score a specific output against a set of examples, ask what the denominator means before you optimize the scorer.\nCauterRule spent two releases blaming a matcher for a number the reference pool was manufacturing. The fix was not a better algorithm. It was recognizing that \"recall\" should mean\nrecall within the problem this rule is about.\nChange the comparison slice and the same model — the same everything — looks 2–3× better and, more importantly, becomes diagnosable.\nReferences\nCauterRule v0.3.0 release notes\nv0.3.0 field test report\nPer-model cloud comparison\nThreshold calibration (§708 / threshold changes)\nPRD — success metrics\nCorpus format spec (domain labels)\nUser guide\n·\nChangelog\nCauterRule v0.3.0 is released.\nThe full recall data and the 40-corpus × 2-model matrix are in the\nfield test report\n. The\nrepo\nis public. Install with\npip install cauterule\n.\nChangelog\n·\nRelease notes",
  "metadata": {
    "title": "Our Recall Was 0.087 and the Model Was Innocent: How Domain-Scoped Replay Doubled It - DEV Community",
    "description": "Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent... Tagged with ai, agents, healthydebate, llm.",
    "keywords": "ai, agents, healthydebate, llm, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/our-recall-was-0087-and-the-model-was-innocent-how-domain-scoped-replay-doubled-it-4ci4",
    "og:title": "Our Recall Was 0.087 and the Model Was Innocent: How Domain-Scoped Replay Doubled It",
    "og:description": "Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8cryvt7oy3g1y3xwd3za.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Our Recall Was 0.087 and the Model Was Innocent: How Domain-Scoped Replay Doubled It",
    "twitter:description": "Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8cryvt7oy3g1y3xwd3za.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/362bf79f472f/code/body-000.txt",
      "chars": 84,
      "language": "text"
    },
    {
      "file": "attachments/362bf79f472f/code/body-001.txt",
      "chars": 291,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/deghosal-2026/CauterRule",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://pypi.org/project/cauterule/",
      "label": "PyPI",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md",
      "label": "v0.3.0 field test report",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/release/v0.3.0/release-notes.md",
      "label": "Release notes",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md",
      "label": "Changelog",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/field-test-results-cloud.md",
      "label": "Per-model cloud comparison",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/threshold-calibration.md",
      "label": "Threshold calibration (§708 / threshold changes)",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/design/prd/07-success-metrics.md",
      "label": "PRD — success metrics",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/corpus/format-spec.md",
      "label": "Corpus format spec (domain labels)",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/USER_GUIDE.md",
      "label": "User guide",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:17Z",
  "source_url": "https://dev.to/debashish_ghosal/our-recall-was-0087-and-the-model-was-innocent-how-domain-scoped-replay-doubled-it-4ci4",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 8328,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/362bf79f472f/pages/source.html",
  "local_html": "attachments/362bf79f472f/pages/article.html",
  "body_base_path": "attachments/362bf79f472f/pages/",
  "body_markdown_path": "attachments/362bf79f472f/pages/article.md"
}