{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "I Let AI Plan 170 Changes. It Made the Same 3 Mistakes Every Time.",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/i-let-ai-plan-170-changes-it-made-the-same-3-mistakes-every-time-33ne",
  "publish_date": "2026-09-17",
  "corpus_stage": "",
  "notes": "",
  "id": "ae2de5cc806e",
  "collected_at": "2026-10-02T14:05:59",
  "title": "I Let AI Plan 170 Changes. It Made the Same 3 Mistakes Every Time. - DEV Community",
  "body_markdown": "Everyone is arguing about which model plans best. I ran 170 goals and found out the model was never the variable. The plan was.\nI built a small engine called\nPlannerCritic\n: one LLM writes a plan, a second LLM reviews it, and a set of deterministic gates decides whether it's allowed to move forward. Then I pointed it at 170 real change-planning goals across 40 domains, including identity management, multi-agent ops, SRE, supply chain policy, and FinOps.\nTotal cost:\n$0.49.\nThe result was not \"the model is bad.\" It was stranger and more useful than that. By the 10th strict goal I noticed the pattern. By the 50th I could predict the failure before the critic printed it. By the 100th I had stopped being surprised and started being annoyed, because the plan was plausible and still wrong.\nThis is the full story: the three defects, why a bigger model didn't help, the deterministic fixes that did, and the numbers from the 170-goal sweep.\nThe setup\nThe architecture is deliberately plain:\nThe planner LLM produces a structured plan: tasks, preconditions, ordering, rollback.\nA deterministic gate layer validates the plan against hard rules.\nA critic LLM reviews what survives the gates.\nIf the critic blocks, the planner revises, bounded by a revision cap.\nIf it can't converge, the engine escalates to a human instead of guessing.\nThe gates are the part that matters. They run with zero LLM calls, in about 4.7 seconds for 1,295 tests, and they are the reason a bad plan never reaches a person dressed up as a good one.\nThe three defects\nEvery strict goal that failed failed for one of three reasons. Not randomly. Not occasionally. Three families.\nUnverified dependencies: 57 blockers.\nThe plan declares something must be true, but no earlier task makes it true. From a model-serving migration:\n[BLOCKER] unverified_dependencies, task=cutover_traffic_100\n  \"Cutover to 100% is dependent on prior traffic stages being\n   established but lacks confirmation of stability before proceeding.\"\nEnter fullscreen mode\nExit fullscreen mode\nThe plan cut over all the traffic. Nothing verified the 10% and 50% stages were healthy.\nUnsafe sequencing: 46 blockers.\nSteps ordered before their prerequisites. From an embedding-index migration:\n[BLOCKER] unsafe_sequencing, task=backfill_vectors\n  \"Backfill operation cannot proceed until the index is verified\n   for quality; it is ordered incorrectly in the sequence.\"\nEnter fullscreen mode\nExit fullscreen mode\nThe migration backfilled vectors before the quality check that should have gated it.\nWeak rollback: 18 blockers.\nHigh-blast-radius steps with a rollback that doesn't actually undo anything. From a multi-tenant database split:\n[BLOCKER] weak_rollback, task=dual_write_setup\n  \"Rollback only switches to single-write mode without addressing\n   the potential inconsistencies dual-write may have introduced.\"\nEnter fullscreen mode\nExit fullscreen mode\nThat's 121 of the 132 concrete blockers in three families. If you're building anything that plans over multiple steps, those three names belong on your monitor:\nunverified, unordered, unrecoverable.\nI tried a bigger model. It didn't help.\nThe instinct is obvious. Use GPT-4o. I tested it both ways: GPT-4o as planner with the small model as critic, and GPT-4o in both roles.\nSame defect pattern. Better prose. Same structural mistakes.\nThat was the moment I stopped blaming model size. The planner wasn't dumb, which is the annoying part. It knew the right steps. It just couldn't close the dependency graph or enforce the ordering. I didn't have a smaller-model problem. I had a planning-structure problem.\nIf you take one thing from this post:\na structural failure in planning is not a parameters problem.\nYou can buy a smarter model and watch it make the same three mistakes in more confident language.\nWhy the revision loop doesn't save you\nThe loop is designed to converge. The critic reports blockers, the planner revises. But the planner tends to fix one blocker and introduce another. It reshuffles task order without closing the dependency gap. It adds rollback to the wrong task.\nAfter a median of\n2 revisions\n, the planner stops making meaningful changes. A convergence detector fires. The engine escalates to a human.\nThe loop worked exactly as designed. The planner was the bottleneck. You cannot prompt your way out of a structural problem.\nTo be fair about where the failure lived: the critic reliably found the same blockers across revisions. The failure was the planner's inability to\nstructurally repair\n, not the critic's judgment. Those are different bugs, and I've hit both.\nThe fix is not more parameters\nThe highest-leverage change was a\nprecondition closer\n: a deterministic linter that runs after the planner drafts and verifies every precondition maps to an earlier task. If a task says \"requires replica_verified,\" some prior task has to produce that fact.\nprecondition\ncloser\n(\nruns\nafter\nplanner\n,\nbefore\ncritic\n):\nfor\nevery\ntask\n.\npreconditions\n:\nassert\nexists\nprior_task\nwhere\nprior_task\n.\nestablishes\n(\np\n)\nelse\n:\nBLOCKER\nunverified_dependencies\nEnter fullscreen mode\nExit fullscreen mode\nOne pass would eliminate\n64 of 132 blockers (48%)\nwithout asking the model to get smarter.\nThe rest needed two more deterministic mechanisms:\nTopological auto-repair (#130)\n: reorders tasks so prerequisites run first, and only surfaces a blocker when the dependency graph is genuinely cyclic.\nOscillation detection (#152)\n: catches the planner cycling between two orderings and terminates the loop early instead of burning revisions.\nIn the v0.2.1 sweep, oscillation detection fired on 5 strict goals that previously would have spun to the revision cap. That's less latency, fewer LLM calls, and a faster escalation to the human who was going to see it anyway.\nNone of this is a model upgrade. All of it is code.\nThe field test arc: from diagnostic to regression gate\nAcross three releases, the field test changed character. It went from finding bugs to proving their absence.\nRelease\nGoals\nCost\nFound by field test\nFound by code review\nv0.1.0\n157\n$0.30\n10\n0\nv0.2.0\n170\n$0.40\n0\n31\nv0.2.1\n170\n$0.49\n0\n10\nv0.1.0 was a diagnostic. It found 10 issues, and only 1 was a traditional failure: 57 of 65 assertion files were in the wrong format and the harness silently returned 0/0. No crash. Just silence.\nBy v0.2.0, code review was finding the bugs before the LLM ever ran. The field test validated fixes instead of discovering them. By v0.2.1, the sweep was a pure regression gate: same 170 goals, diffed against the published baseline,\n30 verdict deltas, all attributable, zero unexplained.\nThe cost went from $0.30 to $0.49. The value went from finding bugs to proving their absence.\nWhat 170 goals actually cost, in operational terms\nThe v0.2.1 sweep added the numbers the community kept asking for:\nMetric\nValue\nLatency (approved) p50\n13.86s\nLatency (escalated) p50\n27.82s\nMean blockers per goal\n2.58\nEscalation decisions per 100 goals\n58.0\nMean LLM calls per goal\n1.4\nMedian revisions to resolution\n1.0\nTwo things surprised me here. First, the median goal resolves in\n1 revision\n. The deterministic precondition closer and topological repair do the ordering work without calling the model at all. Second, an escalated plan takes about twice as long as an approved one, which is the right shape. The system spends its time where the uncertainty is.\n$0.49 for 170 goals plus 60 boundary audits is cheaper than a single developer-hour. The cost was never the reason not to field test. It was the excuse.\nThe counterintuitive result: a maximally non-deterministic critic is safe\nThis is the finding I did not expect.\nI sent the same boundary corpus through the real critic model five times and measured the disagreement. The critic changed its verdict on\nevery trial\nof identical input:\nMetric\nValue\nWhat it means\nlabel_flip_rate\n1.000\nDifferent verdict every trial\nevidence_drift_rate\n1.000\nDifferent explanation every trial\nfamily_migration_rate\n0.000\nNo seeded defect landed in an advisory bucket\nunderclaim_approvals\n0\nNo defective plan got zero blockers\nThe critic is 100% non-deterministic, and it doesn't matter. It never\nunder-claims\na seeded defect. The safety contract doesn't depend on the critic being consistent. It depends on the deterministic gates owning the under-claim direction while code-enforced severity rules own the over-claim direction.\nDeterministic gates catch what must be caught. The LLM critic is allowed to be unstable because it can only add findings, never suppress a gate blocker.\nI tried to inject my own engine\nBecause the safety boundary is code and not prompt, injection attacks didn't move it. Three hand-crafted adversarial goals were blocked. A SWE-bench-derived security oracle blocked\n35/35 flawed variants\nwhile passing 7/7 correct plans, and generated 21 injection traps. Adversarial goals escalated 8/8.\nYou don't secure an LLM system by making the LLM trustworthy. You secure it by making the part that decides sit outside the model.\nThis isn't just my data\nThe literature converges on the same finding. The \"Why Reasoning Fails to Plan\" work (arXiv 2601.22311) shows LLM agents select actions by local evaluation without modeling future consequences; in knowledge-graph traversals, greedy single-step policies hit myopic traps more than 55% of the time. The PlanGenLLMs survey (arXiv 2502.11221) evaluates planning on completeness, executability, optimality, and representation, and finds models consistently fail at\nexecutability\n: preconditions unmet, steps out of order.\nThe field is moving toward hybrids: LLM plus deterministic validation plus classical planning. Not LLM alone.\nWhat I'd tell you if you're building this\nTest your planner on a real corpus. Three demo goals show you nothing. 170 revealed a pattern invisible at small scale.\nMeasure defect\ntypes\n, not pass/fail. If all your failures cluster in one family, you have a specific gap and a specific fix.\nDon't assume a bigger model fixes structural problems. Planning is a reasoning limitation, not a language ability limitation.\nAdd deterministic validation before you trust self-correction. The revision loop is useful. It is not a substitute for structure.\nWatch your gates after they ship. A deterministic gate that silently stops firing makes your metrics look\nbetter\n. The blocker count drops and nobody asks why. I had to add a canary for exactly this.\nThe honest part\nI don't have a planner that reliably closes the dependency graph. I have an engine that\ncatches\nthe failure and escalates. That's a meaningful difference, and I don't want to pretend it's the same thing as solving planning.\nI also don't have proof the fix is complete. The 64-of-132 projection is a projection, not a post-fix measurement. And the whole approach assumes the failure modes are enumerable. The ones I found were, but \"unverified, unordered, unrecoverable\" may not be the full list for domains I haven't tested.\nThe open question I actually care about: can a planner model with explicit graph-state representation close the gap, or is deterministic repair the permanent answer? I don't know yet.\nWhere do you draw the line between \"the model should get this right\" and \"code should catch it\"?\nIf you've shipped agents that plan over multiple steps, I want to know what your planner keeps getting wrong.\nRepo:\nplanner-critic-engine\n·\npip install planner-critic\nPrior posts:\nThe Planner Made the Same 3 Mistakes\n·\nI Ran 170 Agent Goals for $0.49\n·\nThe Gate That Stayed Silent",
  "metadata": {
    "title": "I Let AI Plan 170 Changes. It Made the Same 3 Mistakes Every Time. - DEV Community",
    "description": "Everyone is arguing about which model plans best. I ran 170 goals and found out the model was never... Tagged with ai, agents, llm, testing.",
    "keywords": "ai, agents, llm, testing, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/i-let-ai-plan-170-changes-it-made-the-same-3-mistakes-every-time-33ne",
    "og:title": "I Let AI Plan 170 Changes. It Made the Same 3 Mistakes Every Time.",
    "og:description": "Everyone is arguing about which model plans best. I ran 170 goals and found out the model was never...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frlfujk8j5227d5bprmrz.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "I Let AI Plan 170 Changes. It Made the Same 3 Mistakes Every Time.",
    "twitter:description": "Everyone is arguing about which model plans best. I ran 170 goals and found out the model was never...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frlfujk8j5227d5bprmrz.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/ae2de5cc806e/code/body-000.txt",
      "chars": 193,
      "language": "text"
    },
    {
      "file": "attachments/ae2de5cc806e/code/body-001.txt",
      "chars": 176,
      "language": "text"
    },
    {
      "file": "attachments/ae2de5cc806e/code/body-002.txt",
      "chars": 179,
      "language": "text"
    },
    {
      "file": "attachments/ae2de5cc806e/code/body-003.txt",
      "chars": 190,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/deghosal-2026/planner-critic-engine",
      "label": "PlannerCritic",
      "type": "a"
    },
    {
      "href": "https://dev.to/debashish_ghosal/the-planner-made-the-same-3-mistakes-every-time-a-bigger-model-didnt-fix-it-3170",
      "label": "The Planner Made the Same 3 Mistakes",
      "type": "a"
    },
    {
      "href": "https://dev.to/debashish_ghosal/i-ran-157-agent-goals-for-030-the-field-test-found-10-issues-that-unit-tests-never-would-hgk",
      "label": "I Ran 170 Agent Goals for $0.49",
      "type": "a"
    },
    {
      "href": "https://dev.to/debashish_ghosal/the-gate-that-stayed-silent-when-a-blocker-count-that-drops-reads-as-improvement-3je9",
      "label": "The Gate That Stayed Silent",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:13Z",
  "source_url": "https://dev.to/debashish_ghosal/i-let-ai-plan-170-changes-it-made-the-same-3-mistakes-every-time-33ne",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 11498,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ae2de5cc806e/pages/source.html",
  "local_html": "attachments/ae2de5cc806e/pages/article.html",
  "body_base_path": "attachments/ae2de5cc806e/pages/",
  "body_markdown_path": "attachments/ae2de5cc806e/pages/article.md"
}