{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Tae Kim",
  "title_listed": "Instrument First, Then Prompt: Finding Real Agentic Pipeline Bugs",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/hannune/instrument-first-then-prompt-finding-real-agentic-pipeline-bugs-if9",
  "publish_date": "2026-07-18",
  "corpus_stage": "",
  "notes": "",
  "id": "240ec6d7504c",
  "collected_at": "2026-10-02T14:04:06",
  "title": "Instrument First, Then Prompt: Finding Real Agentic Pipeline Bugs - DEV Community",
  "body_markdown": "The default reaction when an agentic pipeline misbehaves is to open the system prompt and start rewriting it. The instinct makes sense — the prompt is visible, editable, and feels like the thing you control.\nThe problem is that the prompt is almost never where the bug is.\nWhat Actually Breaks\nThree failure modes come up repeatedly in production agentic systems, and none of them are prompt problems.\nChunk boundary truncation.\nThe agent retrieved the right document and extracted the right section. But the chunk ended mid-sentence, cutting off the date field the downstream validation step required. The prompt correctly asked for the date. The agent correctly tried to provide it. The data was not there.\nTool output drift.\nThe agent correctly identified which tool to call. The tool returned a response, but the field name had changed —\nemployee_count\nhad become\nheadcount\nin a dependency update three weeks earlier. The prompt expected\nemployee_count\n. The model interpreted the unexpected field and got it wrong roughly half the time.\nEval dataset blindspot.\nThe agent handled the first four inputs in the eval set correctly. On the fifth, it received a malformed date string the eval set had never included. The agent guessed wrong. The eval passed. Production failed the first time a real user sent that format.\nNone of these are visible in prompt outputs. You cannot find them by reading model responses.\nAdding Instrumentation\nWhen I wire a new agentic pipeline, the first thing I add is not a system prompt improvement — it is instrumentation. Three pieces.\nA decorator around every tool call:\nimport\ntime\nimport\njson\nfrom\nfunctools\nimport\nwraps\ndef\ntraced_tool\n(\ntool_fn\n):\n@wraps\n(\ntool_fn\n)\ndef\nwrapper\n(\n*\nargs\n,\n**\nkwargs\n):\nstart\n=\ntime\n.\ntime\n()\nresult\n=\ntool_fn\n(\n*\nargs\n,\n**\nkwargs\n)\nelapsed\n=\ntime\n.\ntime\n()\n-\nstart\nprint\n(\njson\n.\ndumps\n({\n\"\nevent\n\"\n:\n\"\ntool_call\n\"\n,\n\"\ntool\n\"\n:\ntool_fn\n.\n__name__\n,\n\"\nelapsed_ms\n\"\n:\nround\n(\nelapsed\n*\n1000\n),\n\"\nresult_keys\n\"\n:\nlist\n(\nresult\n.\nkeys\n())\nif\nisinstance\n(\nresult\n,\ndict\n)\nelse\ntype\n(\nresult\n).\n__name__\n}))\nreturn\nresult\nreturn\nwrapper\nEnter fullscreen mode\nExit fullscreen mode\nThe\nresult_keys\nline catches tool output drift. If a field name changes upstream, the log shows the changed shape the next time the tool fires.\nA log call around every context injection:\ndef\nlog_context_injection\n(\ncontext_items\n:\nlist\n,\nstep_name\n:\nstr\n):\nprint\n(\njson\n.\ndumps\n({\n\"\nevent\n\"\n:\n\"\ncontext_inject\n\"\n,\n\"\nstep\n\"\n:\nstep_name\n,\n\"\nitem_count\n\"\n:\nlen\n(\ncontext_items\n),\n\"\nitem_lengths\n\"\n:\n[\nlen\n(\nstr\n(\nitem\n))\nfor\nitem\nin\ncontext_items\n],\n\"\ntotal_chars\n\"\n:\nsum\n(\nlen\n(\nstr\n(\nitem\n))\nfor\nitem\nin\ncontext_items\n)\n}))\nEnter fullscreen mode\nExit fullscreen mode\nThe\nitem_lengths\nlist shows when a chunk is unusually short — which is often the symptom of a boundary cut. A 12-character chunk where you expected 800 characters tells you something different from an empty result.\nA counter around every router branch:\nroute_counter\n:\ndict\n[\nstr\n,\nint\n]\n=\n{}\ndef\nlog_route\n(\nbranch_name\n:\nstr\n):\nroute_counter\n[\nbranch_name\n]\n=\nroute_counter\n.\nget\n(\nbranch_name\n,\n0\n)\n+\n1\nprint\n(\njson\n.\ndumps\n({\n\"\nevent\n\"\n:\n\"\nroute\n\"\n,\n\"\nbranch\n\"\n:\nbranch_name\n,\n\"\ncount\n\"\n:\nroute_counter\n[\nbranch_name\n]\n}))\nEnter fullscreen mode\nExit fullscreen mode\nNone of this requires a tracing backend. Timestamped JSON lines that survive a production deploy are enough to start.\ngrep event=tool_call | jq .\nis sufficient for most initial debugging.\nThe Debugging Loop That Works\nOnce instrumentation is in place, the loop is:\nRun the pipeline on the failing input\nFind the first log entry where something diverged from expectation — a missing key, an unusually short chunk, an unexpected branch\nFix the thing that caused the divergence\nThat thing is almost always one of: a data shape mismatch at a tool boundary, a chunk size configuration that cuts content too early, or an eval dataset that does not cover the input format real users actually send.\nRewriting the prompt to compensate for a broken interface teaches the model to paper over a structural defect. The next upstream change breaks the workaround and you debug it again. Fixing the interface eliminates the failure mode.\nIf You Want a Proper Backend\nFor LangGraph pipelines, LangSmith integrates at the graph level and gives you span-level trace visualization without additional instrumentation code. Weave (Weights and Biases) is the alternative if you are already on the W&B stack. Both capture tool calls, model inputs and outputs, and latency across a full run.\nFor custom pipelines outside any framework, a thin OpenTelemetry layer is roughly an hour of setup and pays for itself the first time you trace a production failure back to a specific tool call.\nCurious what others are using — LangSmith, Weave, custom spans — and whether prompt-first debugging is actually your default when something breaks.",
  "metadata": {
    "title": "Instrument First, Then Prompt: Finding Real Agentic Pipeline Bugs - DEV Community",
    "description": "The default reaction when an agentic pipeline misbehaves is to open the system prompt and start... Tagged with aiagents, python, debugging, observability.",
    "keywords": "aiagents, python, debugging, observability, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/hannune/instrument-first-then-prompt-finding-real-agentic-pipeline-bugs-if9",
    "og:title": "Instrument First, Then Prompt: Finding Real Agentic Pipeline Bugs",
    "og:description": "The default reaction when an agentic pipeline misbehaves is to open the system prompt and start...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1505579962197-df174377e13f%3Fw%3D1200%26h%3D630%26fit%3Dcrop",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@HannuneTech",
    "twitter:title": "Instrument First, Then Prompt: Finding Real Agentic Pipeline Bugs",
    "twitter:description": "The default reaction when an agentic pipeline misbehaves is to open the system prompt and start...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1505579962197-df174377e13f%3Fw%3D1200%26h%3D630%26fit%3Dcrop"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/240ec6d7504c/code/body-000.txt",
      "chars": 544,
      "language": "text"
    },
    {
      "file": "attachments/240ec6d7504c/code/body-001.txt",
      "chars": 336,
      "language": "text"
    },
    {
      "file": "attachments/240ec6d7504c/code/body-002.txt",
      "chars": 272,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:42Z",
  "source_url": "https://dev.to/hannune/instrument-first-then-prompt-finding-real-agentic-pipeline-bugs-if9",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 4893,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/240ec6d7504c/pages/source.html",
  "local_html": "attachments/240ec6d7504c/pages/article.html",
  "body_base_path": "attachments/240ec6d7504c/pages/",
  "body_markdown_path": "attachments/240ec6d7504c/pages/article.md"
}