{
  "source_file": "0930",
  "source_platform": "Ghost",
  "author": "Peng Qian / Data Leads Future",
  "title_listed": "My Agent System Looks Powerful but Is Just Industrial Trash",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://www.dataleadsfuture.com/my-agent-system-looks-powerful-but-is-just-industrial-trash/",
  "publish_date": "2025-12",
  "corpus_stage": "",
  "notes": "",
  "id": "46248de638ac",
  "collected_at": "2026-10-02T12:27:23",
  "title": "My Agent System Looks Powerful but Is Just Industrial Trash",
  "body_markdown": "Agentic AI\nMy Agent System Looks Powerful but Is Just Industrial Trash\nSingle agent, low robustness, and position bias\nPeng Qian\n29 Dec 2025\n—\n8 min read\nShare\nMy Agent System Looks Powerful but Is Just Industrial Trash. Image by DALL-E-3\nThis weekend note is a bit late because Phase One of my Deep Data Analyst project failed for now. That means I can’t continue the promised Data Analyst Agent tutorial.\nWhat Happened?\nI actually built a single-agent data analysis assistant based on the ReAct pattern.\nThis assistant could take a user’s analysis request, come up with a reasonable hypothesis, run EDA and modeling on the uploaded dataset, give professional business insights and actionable suggestions, and even create charts to back up its points.\nIf you’re curious about how it worked, here’s a screenshot that shows how cool it looked:\nThe cool effects of my data analysis agent. Image by Author\nAfter all, this was just a single-agent app. It wasn’t that hard to build. If you remember, I explained how I used a ReAct agent to solve the Advent of Code challenges. Here’s that tutorial:\nHow I Crushed Advent of Code And Solved Hard Problems Using Autogen Jupyter Executor and Qwen3\nA detailed guide on using AI to generate code and solve puzzles automatically and quickly\nData Leads Future\nPeng Qian\nIf you tweak that agent’s prompt a bit, you can get the same kind of data analysis ability I’m talking about.\nWhy Do I Call It a Failure?\nBecause my agent, like most that AI hobbyists build, is just one of those:\nPerfect for impressing your boss with a beautiful, powerful prototype, but once real users try it, it suddenly breaks down and becomes industrial trash.\nWhy Do I Say That?\nMy agent has two serious problems.\n1. Very poor robustness\nThis is the top feedback I got after giving it to analyst users.\nIf you try it once, it looks amazing. It uses methods and technical skills beyond a regular analyst to give you a very professional argument. You’d think replacing humans with AI was the smartest move you've ever made.\nBut data analysis is about testing cause and effect over time. You must run the same analysis daily or weekly to see if the assistant’s advice actually works.\nEven with the same question, the agent changes its hypotheses and analysis methods each run. It then gives different advice each time.\nThat’s what I mean by poor stability and consistency.\nImagine you ask it to use an RFM model to segment your users and give marketing suggestions. Before a campaign, it uses features A, B, C and makes five levels for each. After the campaign, it suddenly adds a derived metric D and now segments on A, B, C, D.\nYou couldn’t even run an A/B test properly.\n2. It suffers from context position bias\nIf you’ve read my earlier posts, you know my Data Analyst agent runs code through a stateful Jupyter Kernel-based interpreter.\nExclusive Reveal: Code Sandbox Tech Behind Manus and Claude Agent Skills\nUse Jupyter code executor to help your agent finish tasks in a smarter way\nData Leads Future\nPeng Qian\nThis lets the agent act like a human analyst, first making a hypothesis, running code in a Jupyter notebook to test it, and then coming up with a new hypothesis based on results — iterating over and over.\nAgents based on the ReAct mode will perform EDA like human analysts. Image by Author\nThis gives the agent strong autonomous exploration and error-recovery skills.\nBut here’s the problem. In a past post, I mentioned that LLMs have position bias when dealing with long conversation histories:\nFixing the Agent Handoff Problem in LlamaIndex’s AgentWorkflow System\nThe position bias in LLMs is the root cause of the problem\nData Leads Future\nPeng Qian\nIn short, LLMs don’t treat each message fairly. They don’t weight importance by recency like you think they would.\nLLMs do not assign weights to message history as people might think. Arxiv 2404.01430\nAs we keep making and testing hypotheses, the history grows. Each message in it matters. The first shows the data structure, a later one proves a hypothesis wrong, so we skip it next time — all important.\nThe LLM doesn’t see it this way. As the process goes, it starts focusing on wrong messages while ignoring the ones that have been fixed. So it repeats mistakes.\nThis either wastes tokens and time or sends the analysis off-track into another topic. Neither is good.\nSo Phase One of my data analysis agent is done.\nAny Ways to Fix It?\nBuild a multi-agent system with atomic skills\nFor robustness, you’d probably think of using a Context Engineer to lock in the plan and metric definitions before analysis starts.\nAlso, when an analysis works well, we should save the plan and prior assumptions in long-term memory.\nBoth mean giving the agent new skills.\nBut remember, my agent is based on ReAct, which means its prompt is already huge — over a thousand lines now.\nAgents based on the ReAct pattern are often too complex to debug. Image by Author\nAdding anything risks breaking this fragile system and disrupting prompt-following.\nSo a single agent won’t cut it. We should split the system into multiple agents with atomic skills, then use some orchestration to bring them together.\nWe can imagine this multi-agent app as a coordinate system with at least these agents:\nIssue Clarification Agent\n— asks the user questions to clarify the problem, confirm metrics, and scope.\nRetrieval Agent\n— pulls metric definitions and calculation methods from a knowledge base, plus analysis methods written by real data scientists.\nPlanner Agent\n— proposes prior hypotheses, sets an analysis approach, and makes a full plan to keep later agents on track.\nAnalyst Agent\n— breaks the plan into steps, uses Python to execute them, and tests the prior hypotheses.\nStoryteller Agent\n— turns complex technical results into engaging business stories and actionable advice for decision-makers.\nValidator Agent\n— ensures the whole process is correct, reliable, and business-compliant.\nOrchestrator Agent\n— manages all the agents and assigns tasks.\nMy new design for the multi-agent data analyst. Image by Author\nChoose the right agent framework\nWe need an agent framework that supports message passing. When a new task comes up or an agent finishes, a message should go to the orchestrator. The orchestrator should also send tasks by message.\nThe framework should support context state saving. Agents’ intermediate results should go to the context, not all to the LLM, so position bias doesn’t get in the way.\nIf you ask GPT, it will recommend LangGraph and Autogen.\nI’d skip LangGraph. Even though its workflow is fine, its agents still run on LangChain, which I just don’t like.\nWhen people compare Autogen with others, they say Autogen is better for research-heavy tasks like data analysis that need more autonomy.\nBut Autogen’s Selector Group Chat, while good for orchestrators, can’t manage message history well. You can’t control what goes to the LLM, and orchestration is a black box.\nAutogen’s GraphFlow is also half-baked. Workflow only supports agent nodes and no context state management.\nI Used Autogen GraphFlow and Qwen3 Coder to Solve Math Problems — And It Worked\nMore reliable than your math professor\nData Leads Future\nPeng Qian\nThe bigger risk: Autogen has stopped development. For a 50k-star agent framework, that’s a shame.\nWhat about Microsoft Agent Framework (MAF)?\nI like it. Easy to use, takes good ideas from earlier frameworks, and avoids their mistakes.\nI’m ready to use it with Qwen3 and DeepSeek:\nMake Microsoft Agent Framework’s Structured Output Work With Qwen and DeepSeek Models\nThings You Always Have to Do When Switching a Framework\nData Leads Future\nPeng Qian\nI’m studying MAF’s Workflow feature now. It’s nice: multiple node types, context state management, OpenTelemetry observability, and orchestration modes like Switch-Case and Multi-Selection. It has almost everything I want.\nIt also feels ambitious. With new abilities like MCP, A2A, AG-UI, and Microsoft backing it, MAF should have a better long-term future than Autogen.\nMy Next Steps\nI’m reading MAF’s user guide and source now. I’ll start using it in my agent system.\nI’m still working on Deep Data Analyst. After switching frameworks, I’ll need to adapt things for a while.\nThe good news: a multi-agent system lets me add skills step by step, so I can share and show progress anytime instead of waiting until the whole project is done. 😂\nI also want to explore Workflow’s potential in MAF. I’ll see if it can handle different AI agent design patterns. That will help us understand how to use this promising framework.\nWhat are you interested in? Leave me a comment.\nDon’t forget to subscribe to my newsletter to get my latest agent research in your inbox without waiting.\nAnd share my blog with your friends — maybe it can help more people.\nFollow Data Leads Future\nOne practical story every month, sharing my hard-learned experiences in the enterprise AI space.\nSubscribe\nEmail sent! Check your inbox to complete your signup.\nYou can unsubscribe anytime.",
  "metadata": {
    "title": "My Agent System Looks Powerful but Is Just Industrial Trash",
    "description": "I built a powerful-looking data analyst AI that wowed in demos but failed in real use. Here’s what went wrong and how I’m fixing it.",
    "keywords": null,
    "author": null,
    "og:site_name": "Data Leads Future",
    "og:type": "article",
    "og:title": "My Agent System Looks Powerful but Is Just Industrial Trash",
    "og:description": "Single agent, low robustness, and position bias",
    "og:url": "https://www.dataleadsfuture.com/my-agent-system-looks-powerful-but-is-just-industrial-trash/",
    "og:image": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/size/w1200/2025/12/cover-1.webp",
    "og:image:width": "1200",
    "og:image:height": "686",
    "twitter:card": "summary_large_image",
    "twitter:title": "My Agent System Looks Powerful but Is Just Industrial Trash",
    "twitter:description": "Single agent, low robustness, and position bias",
    "twitter:url": "https://www.dataleadsfuture.com/my-agent-system-looks-powerful-but-is-just-industrial-trash/",
    "twitter:image": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/size/w1200/2025/12/cover-1.webp",
    "twitter:label1": "Written by",
    "twitter:data1": "Peng Qian",
    "twitter:label2": "Filed under",
    "twitter:data2": "Agentic AI, Career Growth, Generative AI",
    "twitter:site": "@qtalen",
    "twitter:creator": "@qtalen",
    "article:published_time": "2025-12-29T06:49:42.000Z",
    "article:modified_time": "2026-04-22T08:20:05.000Z",
    "article:tag": "Generative AI"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/size/w1200/2025/12/cover-1.webp",
      "bytes": 111212,
      "reused": true,
      "local_path": "attachments/46248de638ac/images/000.webp",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/size/w1200/2025/12/cover-1.webp",
      "alt": "My Agent System Looks Powerful but Is Just Industrial Trash.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/2025/12/data_analysis_report.gif",
      "bytes": 4359783,
      "reused": true,
      "local_path": "attachments/46248de638ac/images/005.gif",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/2025/12/data_analysis_report.gif",
      "alt": "The cool effects of my data analysis agent.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/icon/color_192_192-52.png",
      "bytes": 17721,
      "reused": false,
      "local_path": "attachments/46248de638ac/images/body-69aa670547c52e55d4eb.png",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/icon/color_192_192-52.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/thumbnail/jupyter_executor_cover-1.webp",
      "bytes": 29218,
      "reused": true,
      "local_path": "attachments/46248de638ac/images/007.webp",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/thumbnail/jupyter_executor_cover-1.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/icon/color_192_192-53.png",
      "bytes": 17721,
      "reused": false,
      "local_path": "attachments/46248de638ac/images/body-92a3f8410e64cc190f4a.png",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/icon/color_192_192-53.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/thumbnail/cover-9.webp",
      "bytes": 105414,
      "reused": true,
      "local_path": "attachments/46248de638ac/images/008.webp",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/thumbnail/cover-9.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/2025/12/image-9.png",
      "bytes": 123759,
      "reused": true,
      "local_path": "attachments/46248de638ac/images/009.png",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/2025/12/image-9.png",
      "alt": "Agents based on the ReAct mode will perform EDA like human analysts.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/icon/color_192_192-54.png",
      "bytes": 17721,
      "reused": false,
      "local_path": "attachments/46248de638ac/images/body-4d12c306b82901e3fd70.png",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/icon/color_192_192-54.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/thumbnail/Agents_Hands_Off.webp",
      "bytes": 65104,
      "reused": true,
      "local_path": "attachments/46248de638ac/images/011.webp",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/thumbnail/Agents_Hands_Off.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/2025/12/image-10.png",
      "bytes": 100089,
      "reused": true,
      "local_path": "attachments/46248de638ac/images/012.png",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/2025/12/image-10.png",
      "alt": "LLMs do not assign weights to message history as people might think.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/2025/12/image-11.png",
      "bytes": 82197,
      "reused": true,
      "local_path": "attachments/46248de638ac/images/013.png",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/2025/12/image-11.png",
      "alt": "Agents based on the ReAct pattern are often too complex to debug.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/2025/12/image-12.png",
      "bytes": 16452,
      "reused": true,
      "local_path": "attachments/46248de638ac/images/015.png",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/2025/12/image-12.png",
      "alt": "My new design for the multi-agent data analyst.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/icon/color_192_192-55.png",
      "bytes": 17721,
      "reused": false,
      "local_path": "attachments/46248de638ac/images/body-4b0334090c02ba18b109.png",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/icon/color_192_192-55.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/thumbnail/cover-10.webp",
      "bytes": 68452,
      "reused": true,
      "local_path": "attachments/46248de638ac/images/018.webp",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/thumbnail/cover-10.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/icon/color_192_192-56.png",
      "bytes": 17721,
      "reused": false,
      "local_path": "attachments/46248de638ac/images/body-7ff244f337c704bf1215.png",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/icon/color_192_192-56.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/thumbnail/Agent_Framework_cover-1.webp",
      "bytes": 133426,
      "reused": true,
      "local_path": "attachments/46248de638ac/images/019.webp",
      "src": "https://storage.ghost.io/c/33/67/33678c00-2c15-4961-93e9-497b427e2006/content/images/thumbnail/Agent_Framework_cover-1.webp",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.dataleadsfuture.com/tag/agentic-ai/",
      "label": "Agentic AI",
      "type": "a"
    },
    {
      "href": "https://www.dataleadsfuture.com/author/peng/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.dataleadsfuture.com/how-i-crushed-advent-of-code-and-solved-hard-problems-using-autogen-jupyter-executor-and-qwen3/",
      "label": "How I Crushed Advent of Code And Solved Hard Problems Using Autogen Jupyter Executor and Qwen3 A detailed guide on using",
      "type": "a"
    },
    {
      "href": "https://www.dataleadsfuture.com/exclusive-reveal-code-sandbox-tech-behind-manus-and-claude-agent-skills/",
      "label": "Exclusive Reveal: Code Sandbox Tech Behind Manus and Claude Agent Skills Use Jupyter code executor to help your agent fi",
      "type": "a"
    },
    {
      "href": "https://www.dataleadsfuture.com/fixing-the-agent-handoff-problem-in-llamaindexs-agentworkflow-system/",
      "label": "Fixing the Agent Handoff Problem in LlamaIndex’s AgentWorkflow System The position bias in LLMs is the root cause of the",
      "type": "a"
    },
    {
      "href": "https://www.dataleadsfuture.com/i-used-autogen-graphflow-and-qwen3-coder-to-solve-math-problems-and-it-worked/",
      "label": "I Used Autogen GraphFlow and Qwen3 Coder to Solve Math Problems — And It Worked More reliable than your math professor D",
      "type": "a"
    },
    {
      "href": "https://www.dataleadsfuture.com/make-microsoft-agent-frameworks-structured-output-work-with-qwen-and-deepseek-models/",
      "label": "Make Microsoft Agent Framework’s Structured Output Work With Qwen and DeepSeek Models Things You Always Have to Do When ",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:47Z",
  "source_url": "https://www.dataleadsfuture.com/my-agent-system-looks-powerful-but-is-just-industrial-trash/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 9017,
  "expected_body_images": 16,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/46248de638ac/pages/source.html",
  "local_html": "attachments/46248de638ac/pages/article.html",
  "body_base_path": "attachments/46248de638ac/pages/",
  "body_markdown_path": "attachments/46248de638ac/pages/article.md"
}