{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "I Thought This Was a Classification Problem. It Wasn't.",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/i-thought-this-was-a-classification-problem-it-wasnt-2od9",
  "publish_date": "2026-09-04",
  "corpus_stage": "",
  "notes": "",
  "id": "e2b51b81c874",
  "collected_at": "2026-10-02T14:06:25",
  "title": "I Thought This Was a Classification Problem. It Wasn't. - DEV Community",
  "body_markdown": "Previously:\n9 Bugs That All Looked Like a Working System\n·\nI Built an AI That Rewrites Its Own Prompts\n·\nThe Edit That Fixed 4 Tasks and Broke 1\n·\nI Let an LLM Rewrite Its Own Prompt. The Real Win Was the Gate That Rejected It.\n·\nI Tried 4 Models to Save My Self-Improving Agent. All 4 Failed.\nAgentSelfEdit\nis an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners.\nRepo:\ngithub.com/deghosal-2026/agent-self-edit/tree/v0.3.0\nRelease notes:\ndocs/release/v0.3.0/release-notes.md\nField test report:\ndocs/field-test/v0.3.0/FIELD_TEST_REPORT.md\nFor a while I had a decent excuse for the optimizer looking weak.\nMaybe classification was just a bad place to judge it.\nClassification is rigid. Exact labels. Unforgiving scoring. Easy place for an LLM to look brittle.\nSo when\nv0.3.0\ngot enough infrastructure behind it, I pushed beyond classification. I ran reduced-cost\ncheap-smoke\nloops on extraction, generation, and mixed-domain corpora.\nI expected some pain. I did not expect the same failure to show up three times wearing different clothes.\nThe Cheap-Smoke Runs Were Small, But They Were Enough\nThese were not full-blown multi-iteration field tests on every corpus. The report is careful about that, and I want to be too. The samples were smaller. The goal was narrower.\nI wanted to answer one question:\nis the analyzer's weakness classification-specific, or does it generalize?\nIt generalizes.\nThat answer showed up fast.\nExtraction Looked Reasonable Right Up Until It Didn't\nThe extraction run was almost boring, which is part of what made it useful.\nObserved pattern:\niteration 1:\n0\nup,\n0\ndown\niteration 2:\n1\nup,\n0\ndown,\neffect_size = 0.0455\niteration 3:\n1\nup,\n2\ndown\nThe analyzer wasn't acting crazy. It kept proposing local tweaks around field naming and formatting:\nexact field names\nlowercase conventions\nconcise formatting\noutput consistency\nAll of that sounds normal. That's what makes the result interesting. The analyzer can usually smell a local problem. It just keeps reaching for local instructions as if that alone will carry the day.\nGeneration Was the One That Hurt My Ego a Little\nGeneration exposed something classification only hinted at.\nThe analyzer proposed edits that sounded smart to me when I read them:\nfollow structure more strictly\navoid generic content\nstick to format constraints\nThat is the kind of advice prompt engineers say to each other all the time.\nThen I looked at the results:\niteration 1:\n1\nimproved,\n7\nregressed\niteration 2:\n0\nproposals\niteration 3:\n1\nimproved,\n2\nregressed\nThat was one of the most useful surprises in\nv0.3.0\n.\nGeneration made it painfully obvious that an edit can sound more disciplined and still make the model worse. The prompt got tighter. The assistant got more obedient. The task results got uglier.\nI think a lot of us have done this by hand and talked ourselves into calling it improvement.\nThis time the corpus didn't let me get away with that story.\nMixed-Domain Was the Only One That Gave Me a Little Hope\nMixed-domain was the most interesting non-classification run.\nObserved pattern:\niteration 1:\n0\nup,\n0\ndown\niteration 2:\n0\nup,\n0\ndown\niteration 3:\n2\nup,\n0\ndown,\neffect_size = inf\n,\np = 0.46\nThat third iteration was the first non-classification cheap-smoke result that looked like a real little pocket of signal.\nBut even there, the analyzer needed time to escape its bad habits. The first two iterations were still generic formatting and conciseness advice. Only later did something more domain-specific show up.\nSo mixed-domain gave me two feelings at once: maybe broader corpora help, and no, broader corpora do not magically fix shallow search.\nThe Pattern That Kept Coming Back\nAcross classification, extraction, generation, and mixed-domain, the same structure kept repeating:\nthe analyzer saw a plausible local issue\nit proposed a local wording tweak\nthe tweak sometimes helped a little\nit usually didn't help enough\nsometimes it over-corrected and made things worse\nThat is not four different failures. It is one failure pattern with different surface details.\nIn classification it looked like urgency-boundary rewrites. In extraction it looked like field-format micro-edits. In generation it looked like stricter output discipline. In mixed-domain it started as generic clarity advice.\nSame habit. Different task.\nThat was the big cross-domain learning in\nv0.3.0\n.\nWhy This Surprised Me\nI expected some domains to be better or worse. What I did not expect was for the optimizer's personality to show up this clearly.\nIt has a favorite move: see a plausible local error and patch it with a local wording rule.\nThat is not nothing. It means the analyzer is not random. It is picking up real failure patterns.\nBut it also means the current problem is not really about one corpus or one scorer. The deeper problem is that the analyzer does not explore broadly enough.\nOnce I saw that across corpora, it got a lot harder to blame the wrong thing.\nWhat Worked\nI don't want to oversell the failure here. Some things did work.\nThe framework could actually run across domains. That sounds basic, but earlier versions did not earn that assumption.\nv0.3.0\ndid.\nThe analyzer was usually plausible. The issue was not total nonsense. The issue was narrowness.\nAnd generation gave me one of the sharpest developer lessons in the whole release: stricter prompt wording can absolutely make outputs worse while still sounding like an upgrade.\nWhat I Learned\nCross-domain evidence is a reality check. A lot of AI system stories sound convincing inside one corpus. Once the same behavior starts showing up in extraction, generation, and mixed-domain runs, your favorite explanation gets a lot less special.\nI also learned, again, that \"sounds right\" is a useless quality bar for prompt edits. Generation made that painfully clear.\nGeneration also taught me something specific I did not expect to value this much: it may be the best corpus for catching harmful \"tighten the wording\" edits. Classification is good at showing missed labels and weak boundaries. Generation is better at showing when stricter instructions make the system sound more obedient while actually making outputs worse.\nAnd I learned that the next real frontier here is search breadth. Not just a bigger model. Not just more runs. The analyzer has to get better at escaping its first obvious explanation of the failures.\nWhy Developers Should Care\nIf you're building systems that revise prompts from feedback, don't assume poor results in one domain are just a domain artifact. Run the same loop somewhere with a different failure surface.\nYou might find what I found: the optimizer keeps making the same kind of move, regardless of the task. If that happens, the bug is not in the corpus. The habit is in the optimizer.\nI went into these cross-domain runs hoping they would make the optimizer look better.\nThey did something more useful instead. They made the problem harder to misdiagnose.\nIf you were designing the next corpus for a system like this, would you bias toward rigid tasks like classification, open-ended tasks like generation, or mixed-domain prompts that force broader search? And have you seen an optimizer in your own work keep making the same kind of \"reasonable\" edit over and over, even when the task changes?",
  "metadata": {
    "title": "I Thought This Was a Classification Problem. It Wasn't. - DEV Community",
    "description": "Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own... Tagged with ai, healthydebate, llm, testing.",
    "keywords": "ai, healthydebate, llm, testing, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/i-thought-this-was-a-classification-problem-it-wasnt-2od9",
    "og:title": "I Thought This Was a Classification Problem. It Wasn't.",
    "og:description": "Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2qwk00owc12tgeq9e8zn.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "I Thought This Was a Classification Problem. It Wasn't.",
    "twitter:description": "Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2qwk00owc12tgeq9e8zn.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://dev.to/debashish_ghosal/9-bugs-that-all-looked-like-a-working-system-25mg",
      "label": "9 Bugs That All Looked Like a Working System",
      "type": "a"
    },
    {
      "href": "https://dev.to/debashish_ghosal/i-built-an-ai-that-rewrites-its-own-prompts-its-safety-gate-rejected-every-single-edit-220h",
      "label": "I Built an AI That Rewrites Its Own Prompts",
      "type": "a"
    },
    {
      "href": "https://dev.to/debashish_ghosal/the-edit-that-fixed-4-tasks-and-broke-1-5770",
      "label": "The Edit That Fixed 4 Tasks and Broke 1",
      "type": "a"
    },
    {
      "href": "https://dev.to/debashish_ghosal/i-gave-an-llm-the-keys-to-rewrite-its-own-prompt-then-built-a-gate-that-said-no-4150-times-1h46",
      "label": "I Let an LLM Rewrite Its Own Prompt. The Real Win Was the Gate That Rejected It.",
      "type": "a"
    },
    {
      "href": "https://dev.to/debashish_ghosal/i-tested-4-models-and-none-could-improve-their-own-prompt-the-search-strategy-is-broken-not-the-3ajf",
      "label": "I Tried 4 Models to Save My Self-Improving Agent. All 4 Failed.",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/agent-self-edit/tree/v0.3.0",
      "label": "github.com/deghosal-2026/agent-self-edit/tree/v0.3.0",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/agent-self-edit/blob/v0.3.0/docs/release/v0.3.0/release-notes.md",
      "label": "docs/release/v0.3.0/release-notes.md",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/agent-self-edit/blob/v0.3.0/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md",
      "label": "docs/field-test/v0.3.0/FIELD_TEST_REPORT.md",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:21Z",
  "source_url": "https://dev.to/debashish_ghosal/i-thought-this-was-a-classification-problem-it-wasnt-2od9",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 7358,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/e2b51b81c874/pages/source.html",
  "local_html": "attachments/e2b51b81c874/pages/article.html",
  "body_base_path": "attachments/e2b51b81c874/pages/",
  "body_markdown_path": "attachments/e2b51b81c874/pages/article.md"
}