{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "7 Agent Eval Mistakes That Cost Me Weeks (And the One-Line Fixes That Ended Them)",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/7-agent-eval-mistakes-that-cost-me-weeks-and-the-one-line-fixes-that-ended-them-ho",
  "publish_date": "2026-09-24",
  "corpus_stage": "",
  "notes": "",
  "id": "33811bef9a39",
  "collected_at": "2026-10-02T14:05:55",
  "title": "7 Agent Eval Mistakes That Cost Me Weeks (And the One-Line Fixes That Ended Them) - DEV Community",
  "body_markdown": "You've been there. You run the eval, the number comes back, and something about it doesn't sit right. But the number is the number, so you move on. Three weeks later you find out the number was never the model's fault.\nI spent a week tuning a matcher to explain a 20% pass rate. The bug wasn't in the matcher. It was in the evaluator, in six lines I'd stopped reading because I'd stopped trusting them.\nThis is a list of seven mistakes. I've made all seven, in this order, and each one cost me real days. For each, I'll tell you what I saw, what I tried first (and why it was the wrong layer), and the small change that actually fixed it. They look like seven different problems. They have one root cause: I kept fixing the model instead of the measurement.\nIf\nadd commit push\nis muscle memory but your agent metrics still feel like a black box you're not allowed to open, keep reading.\n1. Trusting a Green Suite\nA green suite is the most expensive lie in software. Mine said 359 tests passed while my agent's real pass rate sat at 20%. Everything was green, so of course I assumed the model was the problem.\nWhat I tried first: more tests. What I should have done: check whether the tests were asserting anything. One of them had a function body of just\npass\n.\ndef\ntest_all_adapters_importable\n():\n# it imported, so it must work, right?\npass\nEnter fullscreen mode\nExit fullscreen mode\npass\nis not a test. It's a green checkmark with a body. In a PlannerCritic review I found 57 of 65 assertion files were in the wrong format, so the harness quietly returned 0/0 and called it a pass. The suite was green because it ran nothing.\nIf a module returns 0/0, treat it as an error, not a pass. Zero assertions is not a quiet success.\n2. Letting the Model Grade Itself\nThis one felt like a model problem. It wasn't.\nThe model scored precision 1.00 and recall 0.02 and still passed. It had found a degenerate trigger it could fire forever and collect the credit. My first instinct was to tune the prompt. That was the wrong layer, and I kept tuning for a day.\nMy 3B model learned to match the literal string\n\"step_1\"\nbecause the reward function paid for overlap, not for correctness. It wasn't learning to review. It was learning what got paid.\nFull write-up here.\nPrecision 1.00 with recall near zero is not a safe model. It's a model that found the cheapest way to look safe.\n3. Scoring Against the Wrong Denominator\nRecall was stuck at 0.087. Same number, two field tests in a row, no matter what I changed. I was about to conclude the model had a ceiling.\nI rebuilt the matcher. That was a waste. The matcher was fine.\nThe real fix was one line of scope: restrict the reference pool to the source domain. The model was answering the right question against a haystack of unrelated gold examples, so it looked bad even when it was mostly right. Once I scoped it, recall went 0.087 to 0.170 / 0.228 with no model change at all.\nThe full numbers are here.\nIf a metric is stuck at the same low value across every model you try, suspect the denominator before you suspect the model.\n4. Optimizing the Layer the Report Named\nA week of matcher work moved the metric ten points. Ten points. I'd been fixing the matcher because that's the layer the failing tests kept naming, and I trusted the report more than I should have.\nThe change that actually moved the needle was six lines in the simulator that generated the failing cases. The score jumped from 20% to 50%. Six lines.\nI wrote about that week here.\nThis is the mistake that stings the most, because the report pointed me at the matcher and I believed it.\nThe layer the failure names is not always the layer that's broken. Prove the origin before you spend the week.\n5. Treating a Model-Independent Failure as a Model Ceiling\nA local 4B and a cloud model hit the exact same wall. Uniform 20% across both. My knee-jerk was to buy a bigger model. Same wall.\nIdentical failure rates across wildly different models is almost never a capability ceiling. It's a shared harness bug. I compared a local 4B, a better cloud model, and role separation, and the results were identical in a way that should have been a warning.\nThe comparison is here.\nIf every model plateaus at the same number, that constant is the thing to fix. Not the model.\n6. Mocks That Can't Break\nUnit tests green. Real agents failing in the field. I wrote more mocks. Wrong move.\nMy mock suite reported 9% pass. The zero-mock field test found every single failure the mocks couldn't express. Why? A mock agent always calls the tool it's told to. A real model, sometimes, just answers in prose.\nA mock can only fail in the ways you already imagined. Real agents fail in the ways you didn't.\nYour mocks are a record of your current imagination, not of the system.\n7. Forgetting the Runner\nA 1,000-run benchmark died at the 80% mark. I blamed the corpus. I re-ran it. It died again.\nThe bug was the runner. One hung API call had discarded 60 completed audits, no error record, nothing. I added a per-trajectory timeout, non-retryable timeout classification, a token cap, quarantine on failure, and cancel-on-shutdown. After that, a clean field test ran 4,768 trajectory runs with zero lost sweeps.\nThe field test report is in the repo.\nA hang is a data-integrity bug, not a performance bug. A partial sweep still looks like data, and that's the trap.\nThe One Thing All Seven Share\nLook back. Only one of these was actually a model problem, and even that one was fixed in the scorer, not the prompt. The other six were measurement bugs: a suite that didn't assert, a denominator that lied, a layer I misidentified, a classifier shared across models, mocks that couldn't fail, and a runner that dropped data.\nI'd rephrase the lesson this way:\nthe reward function is almost always the real bug.\nYou can tune the model forever and the number won't move, because the number was never the model to begin with.\nThe open question I still can't answer: how do you know your eval is honest? I can name the seven mistakes now. I can't fully trust that I've caught the eighth. A green suite with a hidden measurement bug and a red suite with a real bug look identical from the command line. I don't think there's a clean test for that yet, and I'd rather say that out loud than pretend I closed it.\nWhich of these did you ship before it taught you? I'll admit to all seven. Tell me yours.\nCode and receipts:\nagent-eval-forge\n·\nCauterRule\n·\nplanner-critic-engine\n— all MIT, all public.",
  "metadata": {
    "title": "7 Agent Eval Mistakes That Cost Me Weeks (And the One-Line Fixes That Ended Them) - DEV Community",
    "description": "You've been there. You run the eval, the number comes back, and something about it doesn't sit... Tagged with ai, evaluation, llm, programming.",
    "keywords": "ai, evaluation, llm, programming, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/7-agent-eval-mistakes-that-cost-me-weeks-and-the-one-line-fixes-that-ended-them-ho",
    "og:title": "7 Agent Eval Mistakes That Cost Me Weeks (And the One-Line Fixes That Ended Them)",
    "og:description": "You've been there. You run the eval, the number comes back, and something about it doesn't sit...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnymsffvucd9ph0vqdo2p.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "7 Agent Eval Mistakes That Cost Me Weeks (And the One-Line Fixes That Ended Them)",
    "twitter:description": "You've been there. You run the eval, the number comes back, and something about it doesn't sit...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnymsffvucd9ph0vqdo2p.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/33811bef9a39/code/body-000.txt",
      "chars": 88,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://dev.to/debashish_ghosal/my-3b-model-found-a-shortcut-it-took-me-three-fixes-to-close-it-3bec",
      "label": "Full write-up here.",
      "type": "a"
    },
    {
      "href": "https://dev.to/debashish_ghosal/our-recall-was-0087-and-the-model-was-innocent-how-domain-scoped-replay-doubled-it-4ci4",
      "label": "The full numbers are here.",
      "type": "a"
    },
    {
      "href": "https://dev.to/debashish_ghosal/the-6-line-fix-that-outperformed-my-entire-matcher-week-1810",
      "label": "I wrote about that week here.",
      "type": "a"
    },
    {
      "href": "https://dev.to/debashish_ghosal/i-compared-a-local-4b-a-better-cloud-model-and-role-separation-the-results-were-weird-aj8",
      "label": "The comparison is here.",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md",
      "label": "The field test report is in the repo.",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/agent-eval-forge",
      "label": "agent-eval-forge",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule",
      "label": "CauterRule",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/planner-critic-engine",
      "label": "planner-critic-engine",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:12Z",
  "source_url": "https://dev.to/debashish_ghosal/7-agent-eval-mistakes-that-cost-me-weeks-and-the-one-line-fixes-that-ended-them-ho",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 6476,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/33811bef9a39/pages/source.html",
  "local_html": "attachments/33811bef9a39/pages/article.html",
  "body_base_path": "attachments/33811bef9a39/pages/",
  "body_markdown_path": "attachments/33811bef9a39/pages/article.md"
}