{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "My Extraction Score Was 0.08 and the Model Was Innocent: Rebuilding the Ruler",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/my-extraction-score-was-008-and-the-model-was-innocent-rebuilding-the-ruler-2fc1",
  "publish_date": "2026-09-13",
  "corpus_stage": "",
  "notes": "",
  "id": "ad81a1bad1eb",
  "collected_at": "2026-10-02T14:06:10",
  "title": "My Extraction Score Was 0.08 and the Model Was Innocent: Rebuilding the Ruler - DEV Community",
  "body_markdown": "Update — v0.3.1 released.\nCauterRule is now live on\nGitHub\nand\nPyPI\n. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.\npip install cauterule\ngives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The\nv0.3.1 field test report\nevaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the source for every number below.\nRelease notes\n·\nChangelog\nCauterRule\nis an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.\nThe last article ended on an open question:\nat what point does a rule that is right-but-reworded deserve credit, and who decides?\nFor v0.3.1 we built the instrument that question deserves — a direct extraction-accuracy metric, scoring the extracted rule against the\nexpected_rule\nground truth the corpus already carried but we had never used.\nThe first reading was\n0.08\n.\nWe stared at it the way we stared at every low number in this project's history: reach for the model. Then we looked at the ruler. The ruler was broken — in the same way we had accused the replay gate of being broken, which makes this article feel a little like we are grading our own homework on a rubric we just wrote. It is also the most satisfying fix of the release cycle.\nWhat we built\nFour of the 40 corpora carry an\nexpected_rule\nfield — the ground-truth rule a trajectory should produce:\ngolden\n(60, backfilled with the authored scenarios in #735),\nfailures/positive\n(23 trajectories carry one),\nreference-expansion\n(303), and its\nparaphrase-diversity\nslice (15). For every extracted candidate we now compute, per corpus:\nMetric\nComparator\nWhat it grades\ntoken_f1\nliteral token F1, extracted vs\nexpected_rule\nsurface form\nsemantic_f1\nMiniLM cosine on the full rule\nmeaning, whole rule\ndirective_f1\nliteral token F1 on the\ndo\nhalf only\nthe directive phrase\nagreement\ntrigger-only semantic match (after J6)\nthe headline\nThe metric's job is narrow and it matters: it answers\n\"did the model extract the right rule?\"\nindependently of replay, which answers\n\"can we verify it against history?\"\nOne number, two questions — the split from the previous article, finally instrumented.\nThe first reading: 0.08, and why it lied\nThe first version gated\nagreement\non the\ndirective's literal token F1\n. A directive like\n\"pull latest changes before pushing\"\nis a short phrase. A correct rewording —\n\"run a rebase or fetch-merge first\"\n— shares almost no tokens with it and scores ~\n0.22\n. Gate a composite on that, and the headline agreement on\ngolden\ncollapses to\n0.08–0.10\n.\nRead that number naively and the story is \"the model is bad at extracting directives.\" Read the four sub-metrics side by side and the story is different: the trigger — the part of the rule that identifies the failure — was matching at semantic similarity of 0.6+. The model knew\nwhen\n. The gate was punishing it for rephrasing\nwhat to do\n.\nThat is the replay sin from the last article, committed by the new metric: a lexical comparator wearing a validation costume. We had diagnosed the disease in one half of the pipeline and then prescribed the same drug for the other half.\nThe fix (J6): agreement is trigger-only, and the parts are reported, not fused\nTwo changes:\nagreement\nis now the trigger-only semantic match.\nThe directive stops gating the headline; it is reported alongside as\ndirective_f1\nso nobody loses the signal — it just stops\ndeciding\nit.\nEvery sub-metric is reported next to the composite.\nThe spread\nis\nthe finding. A composite that hides its parts cannot be diagnosed; a set of parts can.\nThe regression test that pins this is\ntest_agreement_is_trigger_only_directive_not_gated\nin\ntests/measurement/test_extraction_accuracy.py\n. A metric, once it gates a release, is load-bearing code. It gets tests like anything else.\nTwo smaller honesty fixes in the same commit\nJ10 —\n0.0\nis a value;\nnull\nis a different value.\nThe first version of the metric emitted\nextraction_f1: 0.0\nfor corpora with no\nexpected_rule\nat all (\nraw/ci\n). But 0.0 means\nmeasured, and the answer was zero\n— which on a corpus with no ground truth is a lie. It is indistinguishable from total failure.\nto_dict()\nnow returns\nnull\nwhen\nn == 0\n, and the runner emits\nNone\n. The report can now tell\nno ground truth\nfrom\nground truth said zero\n.\nJ13 — label the rate for what the corpus actually is.\nraw/opencode\nwas reporting a \"false_accept_rate\" of\n0.68 / 0.72\nacross the two models. It is a promotion corpus, not a rejection corpus — that number is the acceptance rate, i.e.\nworking as intended\n. Calling it \"false accept\" makes a pass rate read like a safety violation in every summary file we shipped. Rejection corpora (\nnearmiss\n,\nadversarial/*\n) report\nfalse_accept_rate\n; silence and extraction corpora report\nacceptance_rate\n.\nNeither fix changes a model or a threshold. Both change whether the report\nmeans\nwhat it says.\nThe data\nCloud models, v0.3.1, from the committed artifacts:\nCorpus (n)\nModel\ntoken_f1\nsemantic_f1\ndirective_f1\ntoken_agree\nagreement\ngolden (60)\ngpt-4o-mini\n0.419\n0.611\n0.205\n0.250\n0.783\ngolden (60)\nllama-3.1-8b\n0.484\n0.670\n0.220\n0.367\n0.850\nfailures/positive (23)\ngpt-4o-mini\n0.527\n0.664\n0.302\n0.304\n0.739\nfailures/positive (23)\nllama-3.1-8b\n0.556\n0.652\n0.294\n0.478\n0.783\nreference-expansion (303)\ngpt-4o-mini\n0.608\n0.734\n0.244\n0.548\n0.917\nreference-expansion (303)\nllama-3.1-8b\n0.654\n0.749\n0.205\n0.630\n0.917\nparaphrase-diversity (15)\ngpt-4o-mini\n0.512\n0.718\n0.311\n0.267\n0.933\nparaphrase-diversity (15)\nllama-3.1-8b\n0.509\n0.711\n0.388\n0.200\n0.933\nLook at the spread, not the single column:\nagreement 0.74–0.93\n— the trigger the model extracted is semantically the right trigger most of the time.\ntoken_f1 0.42–0.65\n— in literal wording, the same rule looks mediocre.\ndirective_f1 0.20–0.39\n— the short \"do\" phrase is where literal comparison dies, which is exactly why it should not gate the headline.\nSame model, same corpus, three comparators, three very different stories. The model extracts the\nright rule\nin\ndifferent words\n. That was the v0.3.0 open question (plan Q2) —\n\"is the extraction quality gap the model, or the matcher proxy?\"\n— and this is the empirical answer: the proxy.\nWhat worked\nThe ground truth was already in the corpus.\nexpected_rule\nneeded no new labeling — it was being parsed out and dropped at load. Picking it up is the cheapest high-signal metric we have, and we waited a full release to pick it up.\nReporting the parts, not just the composite.\nThe agreement/token/directive spread is the entire finding. A single fused number would have been another 0.5-something to argue about.\nnull\nas a first-class value.\nDistinguishing\nunmeasurable\nfrom\nzero\nis a one-line change that prevents an entire class of misread reports.\nThe metric got regression tests the day it gated a release decision.\ntest_extraction_accuracy.py\npins the trigger-only definition; J6's behavior is a test, not a comment.\nWhat didn't work\nThe first version of the metric.\nIt was the replay bug reborn: a literal-token gate on the shortest, most freely-reworded part of the rule. A new metric ships with the same assumptions as the old one until someone forces it to look at its own sub-scores.\n0.0\nfor \"not measured.\"\nIt read as total failure in\nsummary.json\nand would have polluted any downstream average silently.\nBorrowed labels.\nfalse_accept_rate\non a promotion corpus is a rate with the wrong sign of connotation. Labels are interpretation; interpret the corpus, not the habit.\nQuestions we still can't answer\nIs trigger-only agreement too generous?\nIf the trigger is right and the directive is nonsense,\nagreement\nsays the extraction was good. Is a right-when/wrong-do rule \"extracted correctly\"? We report\ndirective_f1\nbut do not act on it. That is a decision we have deferred, not solved.\nThe small-n problem.\nfailures/positive\nhas 23 ground-truth trajectories and\nparaphrase-diversity\n15. Agreement at n=15 is a direction, not a claim. The Wilson CIs we now publish for pass rates do not yet cover these extraction numbers.\nDoes\nreference-expansion\n's 0.917 overfit to our authored references?\nThe 303 trajectories are paraphrase-expansions\nof our own reference pool\n. High agreement there partly means \"the model reproduces our phrasings.\" How much of 0.917 transfers to rules nobody in the pool wrote? We do not yet know.\nWhat I learned\nA metric is untested code until it is tested.\nWe wrote a measurement instrument with the same confidence we would write a load-bearing feature — and it shipped a lying first version. The fix was structural (trigger-only, parts reported) but the lesson is procedural: the ruler gets tests before the reading gets believed.\nnull\nis a value, and so is a label.\n0.0\nvs\nnull\nand\nfalse_accept_rate\nvs\nacceptance_rate\nare not cosmetics. A report that cannot distinguish\nunmeasured\nfrom\nzero\n, or\nworking as intended\nfrom\nsafety violation\n, is a report that will be misread by the next person — usually under time pressure, which is when reports are read.\nThe spread between comparators is the diagnosis.\ntoken_f1 0.5, semantic 0.7, directive 0.2 — read together, that triple says exactly where a rewording-resistant rule lives. A single composite says \"medium.\" Composites compress; sub-metrics localize.\nWhen a new metric contradicts the old narrative, suspect the metric first — including the new one.\n0.08 implied the model was bad. 0.78 implied the model was good. Both readings were one comparator choice away from being wrong. The data did not change between the two; the ruler did.\nThe broader lesson\nIf you score a generated artifact against ground truth, you are not adding a number — you are adding a second system with its own bugs, sitting between you and the truth. Test it against cases where you know the answer (a near-verbatim extraction\nmust\nscore high; a reworded-but-correct one\nmust not\ncollapse), publish its parts, and give \"unmeasurable\" its own value. Otherwise you will spend a cycle fixing the thing the metric was designed to exonerate.\nThe honest ruler pointed at the next problem: two corpora had sat at a flat zero for a full release, and the reason was not in the model at all.\nReferences\nCauterRule v0.3.1 release notes\nv0.3.1 field test report (§8 extraction quality, Appendix A J6/J10/J13)\nAuto-generated per-corpus results\nExtraction accuracy tests\nCorpus format spec (\nexpected_rule\n)\nUser guide\n·\nChangelog\nCauterRule v0.3.1 is released.\nThe extraction-accuracy tables, the J6/J10/J13 issue-journal entries, and the full 40-corpus × 2-model sweep are in the\nfield test report\n. The\nrepo\nis public. Install with\npip install cauterule\n.\nChangelog\n·\nRelease notes",
  "metadata": {
    "title": "My Extraction Score Was 0.08 and the Model Was Innocent: Rebuilding the Ruler - DEV Community",
    "description": "Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent... Tagged with ai, llm, measurement, agents.",
    "keywords": "ai, llm, measurement, agents, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/my-extraction-score-was-008-and-the-model-was-innocent-rebuilding-the-ruler-2fc1",
    "og:title": "My Extraction Score Was 0.08 and the Model Was Innocent: Rebuilding the Ruler",
    "og:description": "Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd2hrbeqqhdnko826bi4d.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "My Extraction Score Was 0.08 and the Model Was Innocent: Rebuilding the Ruler",
    "twitter:description": "Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd2hrbeqqhdnko826bi4d.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/deghosal-2026/CauterRule",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://pypi.org/project/cauterule/",
      "label": "PyPI",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.1/FIELD_TEST_REPORT.md",
      "label": "v0.3.1 field test report",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/release/v0.3.1/release-notes.md",
      "label": "Release notes",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/CHANGELOG.md",
      "label": "Changelog",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/tests/measurement/test_extraction_accuracy.py",
      "label": "tests/measurement/test_extraction_accuracy.py",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.1/generated-results.md",
      "label": "Auto-generated per-corpus results",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/corpus/format-spec.md",
      "label": "Corpus format spec ( expected_rule )",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/USER_GUIDE.md",
      "label": "User guide",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:16Z",
  "source_url": "https://dev.to/debashish_ghosal/my-extraction-score-was-008-and-the-model-was-innocent-rebuilding-the-ruler-2fc1",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 10774,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ad81a1bad1eb/pages/source.html",
  "local_html": "attachments/ad81a1bad1eb/pages/article.html",
  "body_base_path": "attachments/ad81a1bad1eb/pages/",
  "body_markdown_path": "attachments/ad81a1bad1eb/pages/article.md"
}