{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "SwirlAI Newsletter\nAurimas Griciūnas",
  "title_listed": "Stop Monitoring AI Systems Like Web Apps",
  "keywords": "AI系统监控, 可观测性, LLMOps, Agent监控, 评估驱动开发",
  "domain": "数据处理",
  "content_type": "实践指南",
  "is_paid_flag": "免费",
  "url": "https://www.newsletter.swirlai.com/p/stop-monitoring-ai-systems-like-web",
  "publish_date": "2026",
  "corpus_stage": "LLMOps/部署",
  "notes": "指出AI系统监控与传统Web应用监控的本质差异，提出Agent时代的新监控范式",
  "id": "a657a99abca0",
  "collected_at": "2026-10-01T20:28:15",
  "title": "Stop Monitoring AI Systems Like Web Services",
  "body_markdown": "👋\nI am\nAurimas\n. I write the SwirlAI Newsletter with the goal of presenting complicated Data related concepts in a simple and easy-to-digest way. My mission is to help You UpSkill and keep You updated on the latest news in AI Engineering, Data Engineering, Machine Learning and overall Data space.\nSwirlAI Newsletter is a reader-supported publication. To receive new posts and support my work, consider becoming a subscriber.\nSubscribe\nMany AI systems are still monitored like the web services they sit next to. The API gateway emits uptime, error rates, and latency percentiles, the dashboards come free with the infrastructure. Unfortunately, none of those numbers can tell you that users stare at a blank screen for four seconds before the first token is displayed, or that token spend per task has doubled since the last prompt update, or that the model has started inventing answers around the retrieved context instead of from it.\nThe gap exists because an LLM system breaks the assumptions web monitoring was built on. Responses are generated token by token, so “latency” is at least three different numbers depending on where on the timeline you stand. Cost scales with tokens rather than requests. Also, the most damaging failures are silent: a quality regression does not throw a 500, it returns confident text with a 200 status code.\nFor me personally it helps to group metrics by the question they answer. Five questions cover most of what goes wrong in production: is it fast, can it scale, is it correct, does it hold up, and when there is an agent in the loop, how does it behave. This article walks through each group, what the metrics mean mechanically, and which ones you have to build yourself because nothing emits them by default.\nThe metrics map\nBefore moving to the questions, next week on Thursday I will be running a workshop - From AI Demo to Deployed App.\nAI engineers can build the backend, but most stop when it’s time to put a usable frontend in front of real users.\nJoin me and learn how to:\nPort a Streamlit prototype or a vibecoded frontend app to v0\nShip the app to production on Vercel\nShip a new UI feature on top of an existing backend\nRegister\nJune 18th, 15:00 GMT\nIs It Fast? (Latency)\nAn LLM request has two phases, and they produce different latency numbers. During prefill the model ingests the entire prompt and builds its internal state, with nothing visible to the user while it happens. During decode the model generates output one token at a time. Every latency metric worth tracking is a position on that timeline.\nTime to first token (TTFT)\nis the amount of time from sending the request to the first token arriving, which is queueing time plus prefill. In a streaming UI this is the number users feel (perceived latency), because it is exactly how long they look at a blank screen. TTFT grows with prompt length, which is why RAG systems that pack large contexts into the prompt pay for it in perceived speed.\nInter-token latency (ITL,\nalso reported as time per output token, TPOT) is the gap between consecutive tokens once streaming starts, and it determines whether output reads as flowing text properly. Users tolerate a slow but steady stream far better than a fast one that freezes.\nEnd-to-end latency at p50, p95, and p99\nis the full span, and output length dominates it. That makes a single global percentile close to meaningless: it averages 50-token classification calls with 2,000-token report generations. Track end-to-end latency per use case, so each number has one workload behind it.\nAgents add a compounding effect. A task that chains several sequential LLM calls multiplies the per-call numbers, and a tolerable per-call p95 can become an intolerable task-level latency. For agentic workloads, set the latency budget at the task level and let it constrain the steps.\nInference latency metrics\nCan It Scale? (Throughput and Cost)\nTokens per second per user versus total system throughput.\nServing systems batch concurrent requests on the same hardware, and the batch size is what you can adjust: larger batches improve total system tokens per second while each individual stream slows down. The two metrics trade off against each other on the same GPUs. So decide per workload which one you are protecting. Interactive chat should favour per-user speed, offline batch processing should favour total throughput.\nInput and output tokens per request.\nThere is not much to add here, it is the core of unit economics, with output tokens typically priced at a multiple of input tokens. Log both per request, break down by use case, monitor carefully.\nCache hit rate.\nPrompt caching makes repeated prompt prefixes dramatically cheaper to process, it is one of the largest single cost levers in systems with stable system prompts, tool definitions, or shared document context. It is also easy to get wrong, because caching matches the prompt prefix byte for byte: a timestamp interpolated near the front of the prompt, or a reordered tool list invalidates the cache. Monitor for falling cache hit rate, it can signal that someone changed how prompts are assembled.\nCost per successful task, not cost per request.\nCost per request makes systems that fail cheaply look good. A request that costs a third as much but succeeds half as often is more expensive where it counts, and failed attempts trigger retries that multiply spend invisibly. This is also the connective metric across all five groups: you can push latency, cost, or quality in isolation, and cost per successful task is the number that tells you whether the system as a whole got better or worse.\nCost per successful task\nSubscribe\nIs It Correct? (Quality)\nNothing in the serving stack emits correctness. Every metric in this group you build yourself.\nTask success rate on a labeled eval set.\nA few dozen to a few hundred examples with known good outcomes, re-run on prompt changes, model upgrade, and retrieval change. This is the regression suite of an AI system. The set does not need to be large to be useful, it needs to be representative and maintained.\nGroundedness for RAG.\nIs the answer supported by the retrieved context, or invented around it? A system can have excellent retrieval and still hallucinate between what was retrieved and what was written. This is becoming a lot less of a problem with powerful models, but in real production systems you eventually want to use the smallest models possible to reduce cost.\nRetrieval precision and recall @ k.\nOf the k chunks retrieved, precision@k asks how many were actually relevant. Recall@k asks how much of the relevant material made it into those k in the first place. Generation cannot fix what retrieval never surfaced, so when answer quality drops, check the retrieval metrics before rewriting prompts. Retrieval is still the harder problem in AI systems.\nLLM-as-judge scores over time, calibrated against human labels.\nJudges make quality measurable at volume, and they drift - a judge model upgrade can change scores with no change in the system being tested. Calibrate against a sample of human labels at the start, and re-calibrate whenever the judge model changes or the score trend looks too good.\nUser feedback signals.\nExplicit thumbs are rare and biased toward the extremes. Regeneration rate and how heavily users edit generated output before accepting it are more often and honest, because they are recorded at the moment the user acts on the output rather than when they feel like rating it.\nQuality measurement for RAG systems\nLearn more about the lifecycle of AI System development utilising Quality metrics in my previous article:\nAgentic AI Flywheels\nAurimas Griciūnas\n·\nMay 27\nRead full story\nDoes It Hold Up? (Reliability)\nError, timeout, and rate-limit rates per provider.\nThe per-provider split matters because aggregate availability hides which dependency is degrading, and rate limits in particular tend to arrive in bursts tied to one provider’s quota.\nRetry and fallback rate.\nFalling back to a backup model keeps availability green while quality quietly changes. Track quality metrics per serving path..\nGuardrail trigger and refusal rates.\nA spike in either means user behavior drifted, an injection attempt might be happening, or a model change moved the refusal boundary. In any case, it’s worth tracking all and it is cheap once the guardrail actions are properly logged.\nGuardrails and model routing\nLearn more about how to measure your AI systems hands-on in my End-to-End AI Engineering Bootcamp.\nKicking off in a week.\nApply code\nLASTCHANCE15\nfor 15% off.\nRegister\nHow Does Your Agent Behave? (Agent Metrics)\nAgents fail in ways single LLM calls will not, because the model makes a sequence of decisions and each one can quietly degrade.\nTool-call error rate, split by cause.\nSchema and argument errors mean the model misunderstands the tool, and the fix is in the tool description. Execution errors mean the tool itself broke, and the fix is in your code.\nSteps and tokens per completed task.\nSuccess rate holding steady while steps per task increase means cost is rising while accuracy does not. Watch this after every model swap and prompt change.\nContext window utilization.\nThe early warning for compaction and truncation issues. Quality usually degrades while the context fills, so a utilization trend tells you to intervene while the failure is still invisible to users.\nI wrote about the state of context engineering in 2026 in one of my latest articles:\nState of Context Engineering in 2026\nAurimas Griciūnas\n·\nMar 22\nRead full story\nLoop detection.\nThe rate at which an agent repeats an identical tool call with identical arguments. A loop is pure token burn, and it is detectable from the trajectory log with a few lines of code.\nThe Instrumentation Gap\nLooking back across the five groups a pattern is clear. Latency and reliability show up on day one, because the gateway, the load balancer, and the provider SDK emit them as a side effect of serving traffic. Quality, cost per task, and agent behavior metrics are not produced until you build the pipeline that measures them: token and cache fields logged from every response, an eval set that runs on every change, a judge with calibration, trajectory logs for agents.\nThis is also why thorough monitoring and real blind spots coexist so often: the monitoring covers the dimensions the infrastructure reports, and the failures sit in the dimensions it does not. The consequence is problems in budgeting. Instrumentation for quality and cost per task belongs in the initial build estimate for an AI feature, not in a later review.\nWhere to Start\nYou should not slow down your development efforts by trying to implement all of the measures at once. An order that provides most information per effort:\nTTFT and end-to-end latency, per use case.\nIf you stream, TTFT is the users perceived latency, both come from timestamps you already have.\nTokens per request and cache hit rate.\nBoth are fields on the API response, logging them is easy, and they make unit economics clearer immediately.\nA labeled eval set with task success rate.\nStart with fifty examples and the discipline to re-run on every relevant change.\nPer-provider error and fallback rates.\nCheap to add, and the fallback-quality interaction is a common silent degradations.\nAgent metrics, when you ship an agent.\nTool-call errors split by cause, steps per task, and context utilizationcan all be found in the trajectory log.\nThis list is representative rather than exhaustive. It deliberately leaves out GPU-level serving metrics (KV cache utilization, batch occupancy) that matter when you host inference yourself (I will cover these in a separate article), embedding and data drift, and business-layer metrics such as containment rate or revenue per session.\nTakeaways\nGroup metrics by the question they answer.\nIs it fast, can it scale, is it correct, does it hold up, how does the agent behave.\nTrack latency as positions on the request timeline.\nTTFT for perceived speed, ITL for stream smoothness, end-to-end per use case. A single global p95 describes none of your workloads.\nMake cost per successful task the connective number.\nIt is the one metric that joins latency, cost, and quality.\nBudget in instrumentation work, not just dashboard work.\nQuality, cost per task, and agent behavior emit nothing by default.\nHope you found the article useful and see you in the next one!\nSubscribe\nPartner with SwirlAI\nShare\nAI Engineering Bootcamp",
  "metadata": {
    "title": "Stop Monitoring AI Systems Like Web Services",
    "description": "Five questions every AI system has to answer, and the metrics that answer them.",
    "keywords": null,
    "author": "Aurimas Griciūnas",
    "og:type": "article",
    "og:title": "Stop Monitoring AI Systems Like Web Services",
    "og:description": "Five questions every AI system has to answer, and the metrics that answer them.",
    "og:image": "https://substackcdn.com/image/fetch/$s_!VROF!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d630c9d-6b9b-4c4f-b845-41e7a243b384_2218x1232.png",
    "og:url": "https://www.newsletter.swirlai.com/p/stop-monitoring-ai-systems-like-web",
    "twitter:title": "Stop Monitoring AI Systems Like Web Services",
    "twitter:description": "Five questions every AI system has to answer, and the metrics that answer them.",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!GCKR!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fswirlai.substack.com%2Fapi%2Fv1%2Fpost_preview%2F201919273%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2026-06-14T08:32:09.511Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!zdwF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7b5b0b-76e1-4abf-8faa-c558ee90d784_1784x1270.png",
      "bytes": 177740,
      "reused": true,
      "local_path": "attachments/a657a99abca0/images/039.png",
      "src": "https://substackcdn.com/image/fetch/$s_!zdwF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7b5b0b-76e1-4abf-8faa-c558ee90d784_1784x1270.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!r284!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a6c5245-a388-40d5-87af-7a821ca5a97f_1741x888.png",
      "bytes": 141239,
      "reused": false,
      "local_path": "attachments/a657a99abca0/images/body-a0d620ef120bdf6504aa.png",
      "src": "https://substackcdn.com/image/fetch/$s_!r284!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a6c5245-a388-40d5-87af-7a821ca5a97f_1741x888.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!-ETP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F074fb432-dda0-4ef6-a2b4-dcc7b47ca9b2_1813x823.png",
      "bytes": 72066,
      "reused": false,
      "local_path": "attachments/a657a99abca0/images/body-0c54a25f0e597e56d155.png",
      "src": "https://substackcdn.com/image/fetch/$s_!-ETP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F074fb432-dda0-4ef6-a2b4-dcc7b47ca9b2_1813x823.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Wjep!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b4d669a-2101-4d60-a035-a28f55f0ddd0_1357x527.png",
      "bytes": 56575,
      "reused": false,
      "local_path": "attachments/a657a99abca0/images/body-e9ac2e6176f0432ba2be.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Wjep!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b4d669a-2101-4d60-a035-a28f55f0ddd0_1357x527.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!kU75!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7ce8817-27d2-42f3-a413-fc9c70c0c497_6876x4679.png",
      "bytes": 3638,
      "reused": false,
      "local_path": "attachments/a657a99abca0/images/body-bc105dd10052f9c11dc3.png",
      "src": "https://substackcdn.com/image/fetch/$s_!kU75!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7ce8817-27d2-42f3-a413-fc9c70c0c497_6876x4679.png",
      "alt": "Agentic AI Flywheels",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Zx-H!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9f2c2fc-ccd4-47bf-8f8c-c33203086d25_1698x464.png",
      "bytes": 43924,
      "reused": false,
      "local_path": "attachments/a657a99abca0/images/body-55b73aeb5a2fe5a71aa4.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Zx-H!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9f2c2fc-ccd4-47bf-8f8c-c33203086d25_1698x464.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!82Y5!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd47b96f2-27c5-4b8b-abb5-12d3494b04f7_5738x3255.png",
      "bytes": 11047,
      "reused": false,
      "local_path": "attachments/a657a99abca0/images/body-a9358b95c3db8eec739f.png",
      "src": "https://substackcdn.com/image/fetch/$s_!82Y5!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd47b96f2-27c5-4b8b-abb5-12d3494b04f7_5738x3255.png",
      "alt": "State of Context Engineering in 2026",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.linkedin.com/in/aurimas-griciunas",
      "label": "Aurimas",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!zdwF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7b5b0b-76e1-4abf-8faa-c558ee90d784_1784x1270.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://maven.com/p/5300f7/from-ai-demo-to-deployed-app",
      "label": "Register",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!r284!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8a6c5245-a388-40d5-87af-7a821ca5a97f_1741x888.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!-ETP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F074fb432-dda0-4ef6-a2b4-dcc7b47ca9b2_1813x823.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!Wjep!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5b4d669a-2101-4d60-a035-a28f55f0ddd0_1357x527.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.newsletter.swirlai.com/p/agentic-ai-flywheels",
      "label": "Agentic AI Flywheels Aurimas Griciūnas · May 27 Read full story",
      "type": "a"
    },
    {
      "href": "https://substack.com/profile/14122259-aurimas-griciunas",
      "label": "Aurimas Griciūnas",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!Zx-H!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9f2c2fc-ccd4-47bf-8f8c-c33203086d25_1698x464.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://maven.com/swirl-ai/end-to-end-ai-engineering",
      "label": "Register",
      "type": "a"
    },
    {
      "href": "https://www.newsletter.swirlai.com/p/state-of-context-engineering-in-2026",
      "label": "State of Context Engineering in 2026 Aurimas Griciūnas · Mar 22 Read full story",
      "type": "a"
    },
    {
      "href": "https://www.newsletter.swirlai.com/p/sponsorships",
      "label": "Partner with SwirlAI",
      "type": "a"
    },
    {
      "href": "https://www.newsletter.swirlai.com/p/agent-skills-progressive-disclosure?utm_source=substack&utm_medium=email&utm_content=share&action=share&token=eyJ1c2VyX2lkIjoxNDEyMjI1OSwicG9zdF9pZCI6MTkwNTE2MzA0LCJpYXQiOjE3NzQxODAzNzIsImV4cCI6MTc3Njc3MjM3MiwiaXNzIjoicHViLTExNDQxNzEiLCJzdWIiOiJwb3N0LXJlYWN0aW9uIn0.LLuKZRByXFbkkcoItKP1zncAuVAOtWLxhe3eJ2wlkwc",
      "label": "Share",
      "type": "a"
    },
    {
      "href": "https://swrlai.com/ai-bootcamp",
      "label": "AI Engineering Bootcamp",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:18Z",
  "source_url": "https://www.newsletter.swirlai.com/p/stop-monitoring-ai-systems-like-web",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 12489,
  "expected_body_images": 7,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/a657a99abca0/pages/source.html",
  "local_html": "attachments/a657a99abca0/pages/article.html",
  "body_base_path": "attachments/a657a99abca0/pages/",
  "body_markdown_path": "attachments/a657a99abca0/pages/article.md"
}