{
  "source_file": "0930",
  "source_platform": "Ghost",
  "author": "Avi Chawla / Daily Dose of Data Science",
  "title_listed": "Rethinking KV Caching For Production Inference",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.dailydoseofds.com/p/rethinking-kv-caching-for-production-inference/",
  "publish_date": "2026-08-17",
  "corpus_stage": "",
  "notes": "",
  "id": "dc188d1f1a0f",
  "collected_at": "2026-10-02T13:45:53",
  "title": "Rethinking KV Caching For Production Inference",
  "body_markdown": "[ Skip to main content  ](https://www.dailydoseofds.com/p/rethinking-kv-caching-for-production-inference/#sx-main)\n[ ![Daily Dose of Data Science](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/12/dailyds-logo-transparent-1.png) ](https://www.dailydoseofds.com)\n  * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n  * [ Archive ](https://www.dailydoseofds.com/archive/)\n  * Guidebooks \n    * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n[ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n[ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n[ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n[ Courses ](https://www.dailydoseofds.com/)\n[ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n[ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n[ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n[ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n[ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n  * Courses \n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n[ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n[ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n[ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n[ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n[ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n[ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n[ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n  *     * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n[ Archive ](https://www.dailydoseofds.com/archive/)\nGuidebooks \n      * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n      * [ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n      * [ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n      * [ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n      * [ Courses ](https://www.dailydoseofds.com/)\n      * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n      * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n      * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n      * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n      * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\nCourses \n      * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n      * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n      * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n      * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n      * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n      * [ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n      * [ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n      * [ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n\n\nSign In  [ Get Started ](https://www.dailydoseofds.com/membership)\n/\n  * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n  * [ Archive ](https://www.dailydoseofds.com/archive/)\n  * Guidebooks \n    * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n    * [ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n    * [ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n    * [ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n    * [ Courses ](https://www.dailydoseofds.com/)\n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n    * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n    * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n    * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n    * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n  * Courses \n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n    * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n    * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n    * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n    * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n    * [ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n    * [ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n    * [ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n  * Sign In \n\n\nJul 7, 2026  LLMOps \n#  Rethinking KV Caching For Production Inference \nA practitioner's guide to KV cache management in production. \n![Avi Chawla](../images/000.jpg)\nAvi Chawla \n👉\n* * *\nResearchers at Stanford studied how AI agents actually spend their inference budgets.\nOne key finding was that ~62% of what gets sent to an agent on every call is just repeated content, i.e., the same system prompts, tool definitions, and documents, which are fed in again and again.\nSo every time the agent takes a single step, you hand it everything from scratch, even if it just processed the exact same info one turn ago:\n![](../images/001.png)\nPer-token prices dropped 80% between 2023 and 2026, with GPT-4 class models falling from 30/Mto30/Mto0.40/M tokens. But agentic workflows consume 5 to 30x more tokens per task than a standard chatbot query, because every step re-sends all that context.\nSo even though each token got cheaper, the total bill went up, since volume outran the price cuts.\nUber shared a similar story recently. After rolling out Claude Code across their engineering org burned through their entire 2026 AI budget in just 4 months. Gartner now forecasts that 40% of AI agent projects will be cancelled by 2027 because of cost overruns alone.\nThe industry is optimizing the wrong variable. Making tokens cheaper doesn’t help if most of those tokens shouldn’t exist in the first place.\nSo, in this article, we’ll learn about [**a new open-source architecture called LMCache**](https://github.com/lmcache/lmcache?ref=dailydoseofds.com) that moves cache management out of the inference engine entirely.\n![](../images/002.gif)\nTeams running it are seeing up to 14x faster time-to-first-token, so understanding it now puts you ahead of nearly everyone running inference today.\n* * *\n#### What happens when we prompt a model in a naive setup\nEvery time you prompt a model, it runs every token through the attention mechanism. For each token, the model computes a Key vector and a Value vector across every attention layer. These vectors capture how the model understands each token’s relationship to every other token in the context.\n![](../images/003.png)\nThis collection of K and V vectors is called the KV cache, and the computation scales quadratically with input length.\n![](../images/004.png)\nOne MI300X GPU generates roughly 15 TB of KV cache per day. Most of it gets thrown away after each request.\nThe KV cache for your system prompt is identical every time you send it. The KV cache for a document you uploaded is identical every time a user asks about it. But the model re-derives that same understanding from scratch, every single time.\nThink of it like re-reading a textbook from page 1 every time someone asks a follow-up about chapter 7. You already understood chapters 1 through 6, but you have no way to save and reuse that understanding.\n* * *\n#### What prefix caching solves and what it doesn’t\nThe industry noticed the above problem and built a technique called prompt caching to deal with it.\nIf two consecutive requests share the same opening tokens (the “prefix”), the provider stores the KV cache from the first request and reuses it on the second. The model skips recomputing those tokens and only processes what’s new.\n![](../images/005.png)\nThis is incredibly helpful. Anthropic's own implementation gives a 90% cost reduction on cached input tokens. Hit rates of 60 to 85% are achievable for stable workloads. For teams with stable system prompts and tool definitions, this is the single highest-leverage optimization available today.\nBut prefix caching has a hard ceiling.\nThe cached portion must be an exact, byte-for-byte prefix of the new request. If you change anything in the cached region (even a single character), it leads to a full cache miss, and it happens in three common scenarios:\n![](../images/006.png)\n  * RAG with multiple documents → You cache document `A` alone and document `B` alone. If a new query needs both documents, the 2nd document’s cached KV state is invalid since it was computed without awareness of the first document.\n  * Document order changes → The same three documents appear in different orders across requests. Every permutation is a cache miss, even though the documents themselves are identical.\n  * Growing conversation history → Each new turn changes the full context after the prefix. Earlier cached states beyond the stable prefix become useless.\n\n![](../images/007.png)\nAlibaba Cloud’s production data validates these limitations. 10% of KV cache blocks serve 77% of all hits. Most cached content never gets reused because the rigid prefix-matching rule prevents it.\nPrefix caching is a meaningful optimization, but it only helps when your context has a long, unchanging beginning, and many real-world workloads don’t look like that.\n* * *\n#### Another performance bottleneck with caching\nEvery KV cache library runs inside the inference engine’s process. That means cache operations (storing, loading, moving KV tensors around) and the actual inference computation share the same resources.\nThey can’t run at the same time, so when the engine is busy managing cache, it stops doing inference, and vice versa.\n![Image](../images/008.jpeg)\nGoogle's TurboQuant shows this effect. It is a recent KV cache quantization technique that compresses the cache to 3 bits per value with zero accuracy loss. But when it runs inside the inference engine, it causes 20%+ inference slowdown.\nCache management and inference serving are fundamentally different workloads.\nOne is I/O-heavy (moving large tensors between GPU, CPU, and storage). The other is compute-heavy (matrix multiplications on GPU).\n* * *\n#### LMCache and the disaggregated approach\nLMCache is an [**open-source project**](https://github.com/lmcache/lmcache?ref=dailydoseofds.com) (10k+ stars) that takes a fundamentally different approach. Instead of running cache management inside the inference engine, it runs as a completely separate process alongside it.\n[LMCache GitHub Repo](https://github.com/lmcache/lmcache?ref=dailydoseofds.com)\n![](../images/009.png)\nIn practice, LMCache connects to the inference engine through shared GPU memory. The engine just tells LMCache “here are the block IDs I need” (tiny messages, almost no data).\nAll the heavy work of actually moving KV tensors between GPU, CPU, and storage happens inside LMCache’s own process. The inference engine doesn’t even notice it’s happening.\nThis separation produces three benefits:\n![](../images/010.png)\n  * No resource contention → Cache I/O never blocks inference, and inference never blocks cache I/O. The 20% throughput loss from running optimization techniques inside the engine disappears.\n  * Zero-copy sharing across GPUs → In the traditional setup, sharing cached data between two GPUs requires multiple memory copies. LMCache lets both GPUs read and write the same memory region directly, skipping those copies entirely.\n  * Multi-tier parallel loading → Cached data can live across GPU memory, CPU RAM, local SSD, and remote storage. Traditional approaches check these one by one, bottlenecking on the slowest tier. LMCache checks all of them simultaneously and streams data from wherever it finds a match, in parallel.\n\n\n> The performance difference is significant. On H200 GPUs with the Qwen3-235B model and 50 concurrent users, LMCache delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. Startup time drops from over 3 minutes to about 30 seconds.\nAlso, LMCache integrates with all major inference engines (vLLM, SGLang, TensorRT-LLM) and supports both NVIDIA and AMD GPUs.\n* * *\n#### Solving the prefix problem with CacheBlend\nLMCache’s architecture solves the performance side of caching.\nBut recall another problem we discussed above, where a query needed 2 documents.\nThe LMCache team’s research paper CacheBlend, which won the EuroSys 2025 Best Paper Award, directly addresses this limitation.\nThe observation is that in modern transformer models, most tokens primarily attend to their own local context. Only a small fraction of tokens have strong connections across document boundaries.\nCacheBlend exploits this by identifying just those few tokens and selectively recomputing only them. Everything else gets reused as-is from the independent caches.\n![](../images/011.png)\nThis gives 2 to 4x faster processing for multi-document queries (the kind you see in RAG apps) without any quality loss. Instead of recomputing everything from scratch when documents are combined, CacheBlend recovers the missing cross-document understanding at a fraction of the cost.\n* * *\n#### Using LMCache in production\n[**LMCache**](https://github.com/lmcache/lmcache?ref=dailydoseofds.com) isn’t a research prototype but rather ships with the infrastructure that production teams expect.\n  * Prometheus and OpenTelemetry integration for tracking cache hit rates and I/O performance.\n  * Kubernetes operator for deployment\n  * CLI for debugging and benchmarking.\n\n\nIf the inference engine crashes, LMCache preserves all cached data on CPU and storage, so recovery doesn’t start cold.\nIf LMCache itself crashes, the inference engine enters a downgrade mode where caching is disabled, but inference continues normally, and it reconnects automatically when the cache process recovers.\nNeither failure takes the whole system down.\n[**You can find the LMCache GitHub repo here →**](https://github.com/lmcache/lmcache?ref=dailydoseofds.com)\n[LMCache GitHub Repo](https://github.com/lmcache/lmcache?ref=dailydoseofds.com)\n**(don’t forget to star it ⭐️)**\nGood day!\nPublished on Jul 7, 2026 \nComments \nShare \nCopy link [ Share to X ](https://x.com/intent/tweet?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Frethinking-kv-caching-for-production-inference%2F&text=Rethinking%20KV%20Caching%20For%20Production%20Inference) [ Share to Facebook ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Frethinking-kv-caching-for-production-inference%2F) [ Share to Linkedin ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Frethinking-kv-caching-for-production-inference%2F)\nCopied \n[ Previous  How to Reduce LLM Costs by 50-60% Using Model Routing  ](https://www.dailydoseofds.com/p/how-to-reduce-llm-costs-by-50-60-using-model-routing/) [ Next  Agentic RL: Environments, Trajectories, and the Training  ](https://www.dailydoseofds.com/rl-course-part-12/)\n[ ![Daily Dose of Data Science](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/12/dailyds-logo-transparent-1.png) ](https://www.dailydoseofds.com)\nA daily column with insights, observations, tutorials and best practices on python and data science. Read by industry professionals at big tech, startups, and engineering students. \n  * Menu \n  * [ Contact ](https://www.dailydoseofds.com/contact/)\n  * [ FAQ ](https://www.dailydoseofds.com/faq/)\n\n\n[Daily Dose of Data Science](https://www.dailydoseofds.com) © 2026 \n[ ](https://x.com/_avichawla \"X \\(Formerly Twitter\\)\")\n",
  "metadata": {
    "title": "Rethinking KV Caching For Production Inference",
    "description": "A practitioner's guide to KV cache management in production.",
    "keywords": null,
    "author": null,
    "og:site_name": "Daily Dose of Data Science",
    "og:type": "article",
    "og:title": "Rethinking KV Caching For Production Inference",
    "og:description": "A practitioner's guide to KV cache management in production.",
    "og:url": "https://www.dailydoseofds.com/p/rethinking-kv-caching-for-production-inference/",
    "og:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/lm-cache-visual.gif",
    "og:image:width": "1200",
    "og:image:height": "850",
    "twitter:card": "summary_large_image",
    "twitter:title": "Rethinking KV Caching For Production Inference",
    "twitter:description": "A practitioner's guide to KV cache management in production.",
    "twitter:url": "https://www.dailydoseofds.com/p/rethinking-kv-caching-for-production-inference/",
    "twitter:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/lm-cache-visual.gif",
    "twitter:label1": "Written by",
    "twitter:data1": "Avi Chawla",
    "twitter:label2": "Filed under",
    "twitter:data2": "LLMOps",
    "twitter:site": "@_avichawla",
    "article:published_time": "2026-07-06T20:53:00.000Z",
    "article:modified_time": "2026-08-17T20:54:19.000Z",
    "article:tag": "LLMOps"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "alt": "Avi Chawla",
      "local_path": "attachments/dc188d1f1a0f/images/000.jpg",
      "bytes": 11344,
      "downloaded": true,
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "reused": true,
      "retried_at": "2026-10-18T17:30:46Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!BIUd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebb39b4-9521-4593-8c33-50617cad14f3_1200x900.png",
      "alt": "",
      "local_path": "attachments/dc188d1f1a0f/images/001.png",
      "bytes": 317700,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!BIUd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebb39b4-9521-4593-8c33-50617cad14f3_1200x900.png",
      "reused": true,
      "retried_at": "2026-10-18T17:30:46Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!fFcv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3469bd3a-08a1-4389-89fb-59434b6cb71c_960x671.gif",
      "alt": "",
      "local_path": "attachments/dc188d1f1a0f/images/002.gif",
      "bytes": 699288,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!fFcv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3469bd3a-08a1-4389-89fb-59434b6cb71c_960x671.gif",
      "reused": true,
      "retried_at": "2026-10-18T17:30:46Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!VbSm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b4b874e-c487-4872-a9df-eb59252341ba_1200x900.png",
      "alt": "",
      "local_path": "attachments/dc188d1f1a0f/images/003.png",
      "bytes": 386649,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!VbSm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b4b874e-c487-4872-a9df-eb59252341ba_1200x900.png",
      "reused": true,
      "retried_at": "2026-10-18T17:30:46Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!hDhz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F098550ae-41d0-4c13-83ab-a84e8d010185_1200x800.png",
      "alt": "",
      "local_path": "attachments/dc188d1f1a0f/images/004.png",
      "bytes": 171261,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!hDhz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F098550ae-41d0-4c13-83ab-a84e8d010185_1200x800.png",
      "reused": true,
      "retried_at": "2026-10-18T17:30:46Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!tjF8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5560c971-cd1f-4e6c-babd-2ca69a6d94fe_1024x565.png",
      "alt": "",
      "local_path": "attachments/dc188d1f1a0f/images/005.png",
      "bytes": 124364,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!tjF8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5560c971-cd1f-4e6c-babd-2ca69a6d94fe_1024x565.png",
      "reused": true,
      "retried_at": "2026-10-18T17:31:26Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!jhH-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b235653-a216-4f9d-a696-0bc3d1849358_1200x675.png",
      "alt": "",
      "local_path": "attachments/dc188d1f1a0f/images/006.png",
      "bytes": 303259,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!jhH-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b235653-a216-4f9d-a696-0bc3d1849358_1200x675.png",
      "reused": true,
      "retried_at": "2026-10-18T17:31:26Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!asa0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01ff49cb-bcea-4f62-a4f2-7f5aeadc2ddc_1024x559.png",
      "alt": "",
      "local_path": "attachments/dc188d1f1a0f/images/007.png",
      "bytes": 154253,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!asa0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01ff49cb-bcea-4f62-a4f2-7f5aeadc2ddc_1024x559.png",
      "reused": true,
      "retried_at": "2026-10-18T17:31:26Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!-F2O!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce860fb-5781-405c-ac3b-a7aa6afa5a0e_1200x675.jpeg",
      "alt": "Image",
      "local_path": "attachments/dc188d1f1a0f/images/008.jpeg",
      "bytes": 67237,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!-F2O!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce860fb-5781-405c-ac3b-a7aa6afa5a0e_1200x675.jpeg",
      "reused": true,
      "retried_at": "2026-10-18T17:31:26Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!PWy0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4d68035-dc85-47a2-807e-ae3019333712_2397x1333.png",
      "alt": "",
      "local_path": "attachments/dc188d1f1a0f/images/009.png",
      "bytes": 136536,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!PWy0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4d68035-dc85-47a2-807e-ae3019333712_2397x1333.png",
      "reused": true,
      "retried_at": "2026-10-18T17:31:26Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!Ii23!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf6a69a5-e245-4987-b37a-7fcd52a3a3bb_1200x800.png",
      "alt": "",
      "local_path": "attachments/dc188d1f1a0f/images/010.png",
      "bytes": 274369,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Ii23!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf6a69a5-e245-4987-b37a-7fcd52a3a3bb_1200x800.png",
      "reused": true,
      "retried_at": "2026-10-18T17:31:26Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!PY8R!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80e0eaa8-329d-4687-a3bc-62a29400170c_1200x675.png",
      "alt": "",
      "local_path": "attachments/dc188d1f1a0f/images/011.png",
      "bytes": 244266,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!PY8R!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80e0eaa8-329d-4687-a3bc-62a29400170c_1200x675.png",
      "reused": true,
      "retried_at": "2026-10-18T17:31:26Z"
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/p/rethinking-kv-caching-for-production-inference/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/archive/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/what-is-mcp/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/mlops-crash-course-part-1/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/rl-course-part-1/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/membership"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/p/how-to-reduce-llm-costs-by-50-60-using-model-routing/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/rl-course-part-12/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/contact/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/faq/"
    },
    {
      "type": "external",
      "href": "https://dailydoseofds.github.io/ai-engg-book/"
    },
    {
      "type": "external",
      "href": "https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view"
    },
    {
      "type": "external",
      "href": "https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view"
    },
    {
      "type": "external",
      "href": "https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view"
    },
    {
      "type": "external",
      "href": "https://github.com/lmcache/lmcache"
    },
    {
      "type": "external",
      "href": "https://x.com/intent/tweet?text=Rethinking+KV+Caching+For+Production+Inference&url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Frethinking-kv-caching-for-production-inference%2F"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Frethinking-kv-caching-for-production-inference%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Frethinking-kv-caching-for-production-inference%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/_avichawla"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:32:37Z",
  "source_url": "https://www.dailydoseofds.com/p/rethinking-kv-caching-for-production-inference/",
  "delivery_status": "unverified",
  "discovery_status": "short_or_dynamic",
  "missing_reason": "正文过短或动态页面，未确认完整",
  "source_body_markdown": "[ Skip to main content  ](https://www.dailydoseofds.com/p/rethinking-kv-caching-for-production-inference/#sx-main)\n[ ![Daily Dose of Data Science](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/12/dailyds-logo-transparent-1.png) ](https://www.dailydoseofds.com)\n  * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n  * [ Archive ](https://www.dailydoseofds.com/archive/)\n  * Guidebooks \n    * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n[ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n[ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n[ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n[ Courses ](https://www.dailydoseofds.com/)\n[ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n[ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n[ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n[ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n[ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n  * Courses \n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n[ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n[ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n[ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n[ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n[ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n[ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n[ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n  *     * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n[ Archive ](https://www.dailydoseofds.com/archive/)\nGuidebooks \n      * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n      * [ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n      * [ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n      * [ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n      * [ Courses ](https://www.dailydoseofds.com/)\n      * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n      * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n      * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n      * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n      * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\nCourses \n      * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n      * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n      * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n      * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n      * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n      * [ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n      * [ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n      * [ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n\n\nSign In  [ Get Started ](https://www.dailydoseofds.com/membership)\n/\n  * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n  * [ Archive ](https://www.dailydoseofds.com/archive/)\n  * Guidebooks \n    * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n    * [ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n    * [ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n    * [ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n    * [ Courses ](https://www.dailydoseofds.com/)\n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n    * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n    * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n    * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n    * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n  * Courses \n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n    * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n    * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n    * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n    * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n    * [ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n    * [ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n    * [ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n  * Sign In \n\n\nJul 7, 2026  LLMOps \n#  Rethinking KV Caching For Production Inference \nA practitioner's guide to KV cache management in production. \n![Avi Chawla](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg)\nAvi Chawla \n👉\n* * *\nResearchers at Stanford studied how AI agents actually spend their inference budgets.\nOne key finding was that ~62% of what gets sent to an agent on every call is just repeated content, i.e., the same system prompts, tool definitions, and documents, which are fed in again and again.\nSo every time the agent takes a single step, you hand it everything from scratch, even if it just processed the exact same info one turn ago:\n![](https://substackcdn.com/image/fetch/$s_!BIUd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ebb39b4-9521-4593-8c33-50617cad14f3_1200x900.png)\nPer-token prices dropped 80% between 2023 and 2026, with GPT-4 class models falling from 30/Mto30/Mto0.40/M tokens. But agentic workflows consume 5 to 30x more tokens per task than a standard chatbot query, because every step re-sends all that context.\nSo even though each token got cheaper, the total bill went up, since volume outran the price cuts.\nUber shared a similar story recently. After rolling out Claude Code across their engineering org burned through their entire 2026 AI budget in just 4 months. Gartner now forecasts that 40% of AI agent projects will be cancelled by 2027 because of cost overruns alone.\nThe industry is optimizing the wrong variable. Making tokens cheaper doesn’t help if most of those tokens shouldn’t exist in the first place.\nSo, in this article, we’ll learn about [**a new open-source architecture called LMCache**](https://github.com/lmcache/lmcache?ref=dailydoseofds.com) that moves cache management out of the inference engine entirely.\n![](https://substackcdn.com/image/fetch/$s_!fFcv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3469bd3a-08a1-4389-89fb-59434b6cb71c_960x671.gif)\nTeams running it are seeing up to 14x faster time-to-first-token, so understanding it now puts you ahead of nearly everyone running inference today.\n* * *\n#### What happens when we prompt a model in a naive setup\nEvery time you prompt a model, it runs every token through the attention mechanism. For each token, the model computes a Key vector and a Value vector across every attention layer. These vectors capture how the model understands each token’s relationship to every other token in the context.\n![](https://substackcdn.com/image/fetch/$s_!VbSm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b4b874e-c487-4872-a9df-eb59252341ba_1200x900.png)\nThis collection of K and V vectors is called the KV cache, and the computation scales quadratically with input length.\n![](https://substackcdn.com/image/fetch/$s_!hDhz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F098550ae-41d0-4c13-83ab-a84e8d010185_1200x800.png)\nOne MI300X GPU generates roughly 15 TB of KV cache per day. Most of it gets thrown away after each request.\nThe KV cache for your system prompt is identical every time you send it. The KV cache for a document you uploaded is identical every time a user asks about it. But the model re-derives that same understanding from scratch, every single time.\nThink of it like re-reading a textbook from page 1 every time someone asks a follow-up about chapter 7. You already understood chapters 1 through 6, but you have no way to save and reuse that understanding.\n* * *\n#### What prefix caching solves and what it doesn’t\nThe industry noticed the above problem and built a technique called prompt caching to deal with it.\nIf two consecutive requests share the same opening tokens (the “prefix”), the provider stores the KV cache from the first request and reuses it on the second. The model skips recomputing those tokens and only processes what’s new.\n![](https://substackcdn.com/image/fetch/$s_!tjF8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5560c971-cd1f-4e6c-babd-2ca69a6d94fe_1024x565.png)\nThis is incredibly helpful. Anthropic's own implementation gives a 90% cost reduction on cached input tokens. Hit rates of 60 to 85% are achievable for stable workloads. For teams with stable system prompts and tool definitions, this is the single highest-leverage optimization available today.\nBut prefix caching has a hard ceiling.\nThe cached portion must be an exact, byte-for-byte prefix of the new request. If you change anything in the cached region (even a single character), it leads to a full cache miss, and it happens in three common scenarios:\n![](https://substackcdn.com/image/fetch/$s_!jhH-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b235653-a216-4f9d-a696-0bc3d1849358_1200x675.png)\n  * RAG with multiple documents → You cache document `A` alone and document `B` alone. If a new query needs both documents, the 2nd document’s cached KV state is invalid since it was computed without awareness of the first document.\n  * Document order changes → The same three documents appear in different orders across requests. Every permutation is a cache miss, even though the documents themselves are identical.\n  * Growing conversation history → Each new turn changes the full context after the prefix. Earlier cached states beyond the stable prefix become useless.\n\n![](https://substackcdn.com/image/fetch/$s_!asa0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01ff49cb-bcea-4f62-a4f2-7f5aeadc2ddc_1024x559.png)\nAlibaba Cloud’s production data validates these limitations. 10% of KV cache blocks serve 77% of all hits. Most cached content never gets reused because the rigid prefix-matching rule prevents it.\nPrefix caching is a meaningful optimization, but it only helps when your context has a long, unchanging beginning, and many real-world workloads don’t look like that.\n* * *\n#### Another performance bottleneck with caching\nEvery KV cache library runs inside the inference engine’s process. That means cache operations (storing, loading, moving KV tensors around) and the actual inference computation share the same resources.\nThey can’t run at the same time, so when the engine is busy managing cache, it stops doing inference, and vice versa.\n![Image](https://substackcdn.com/image/fetch/$s_!-F2O!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ce860fb-5781-405c-ac3b-a7aa6afa5a0e_1200x675.jpeg)\nGoogle's TurboQuant shows this effect. It is a recent KV cache quantization technique that compresses the cache to 3 bits per value with zero accuracy loss. But when it runs inside the inference engine, it causes 20%+ inference slowdown.\nCache management and inference serving are fundamentally different workloads.\nOne is I/O-heavy (moving large tensors between GPU, CPU, and storage). The other is compute-heavy (matrix multiplications on GPU).\n* * *\n#### LMCache and the disaggregated approach\nLMCache is an [**open-source project**](https://github.com/lmcache/lmcache?ref=dailydoseofds.com) (10k+ stars) that takes a fundamentally different approach. Instead of running cache management inside the inference engine, it runs as a completely separate process alongside it.\n[LMCache GitHub Repo](https://github.com/lmcache/lmcache?ref=dailydoseofds.com)\n![](https://substackcdn.com/image/fetch/$s_!PWy0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4d68035-dc85-47a2-807e-ae3019333712_2397x1333.png)\nIn practice, LMCache connects to the inference engine through shared GPU memory. The engine just tells LMCache “here are the block IDs I need” (tiny messages, almost no data).\nAll the heavy work of actually moving KV tensors between GPU, CPU, and storage happens inside LMCache’s own process. The inference engine doesn’t even notice it’s happening.\nThis separation produces three benefits:\n![](https://substackcdn.com/image/fetch/$s_!Ii23!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf6a69a5-e245-4987-b37a-7fcd52a3a3bb_1200x800.png)\n  * No resource contention → Cache I/O never blocks inference, and inference never blocks cache I/O. The 20% throughput loss from running optimization techniques inside the engine disappears.\n  * Zero-copy sharing across GPUs → In the traditional setup, sharing cached data between two GPUs requires multiple memory copies. LMCache lets both GPUs read and write the same memory region directly, skipping those copies entirely.\n  * Multi-tier parallel loading → Cached data can live across GPU memory, CPU RAM, local SSD, and remote storage. Traditional approaches check these one by one, bottlenecking on the slowest tier. LMCache checks all of them simultaneously and streams data from wherever it finds a match, in parallel.\n\n\n> The performance difference is significant. On H200 GPUs with the Qwen3-235B model and 50 concurrent users, LMCache delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. Startup time drops from over 3 minutes to about 30 seconds.\nAlso, LMCache integrates with all major inference engines (vLLM, SGLang, TensorRT-LLM) and supports both NVIDIA and AMD GPUs.\n* * *\n#### Solving the prefix problem with CacheBlend\nLMCache’s architecture solves the performance side of caching.\nBut recall another problem we discussed above, where a query needed 2 documents.\nThe LMCache team’s research paper CacheBlend, which won the EuroSys 2025 Best Paper Award, directly addresses this limitation.\nThe observation is that in modern transformer models, most tokens primarily attend to their own local context. Only a small fraction of tokens have strong connections across document boundaries.\nCacheBlend exploits this by identifying just those few tokens and selectively recomputing only them. Everything else gets reused as-is from the independent caches.\n![](https://substackcdn.com/image/fetch/$s_!PY8R!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80e0eaa8-329d-4687-a3bc-62a29400170c_1200x675.png)\nThis gives 2 to 4x faster processing for multi-document queries (the kind you see in RAG apps) without any quality loss. Instead of recomputing everything from scratch when documents are combined, CacheBlend recovers the missing cross-document understanding at a fraction of the cost.\n* * *\n#### Using LMCache in production\n[**LMCache**](https://github.com/lmcache/lmcache?ref=dailydoseofds.com) isn’t a research prototype but rather ships with the infrastructure that production teams expect.\n  * Prometheus and OpenTelemetry integration for tracking cache hit rates and I/O performance.\n  * Kubernetes operator for deployment\n  * CLI for debugging and benchmarking.\n\n\nIf the inference engine crashes, LMCache preserves all cached data on CPU and storage, so recovery doesn’t start cold.\nIf LMCache itself crashes, the inference engine enters a downgrade mode where caching is disabled, but inference continues normally, and it reconnects automatically when the cache process recovers.\nNeither failure takes the whole system down.\n[**You can find the LMCache GitHub repo here →**](https://github.com/lmcache/lmcache?ref=dailydoseofds.com)\n[LMCache GitHub Repo](https://github.com/lmcache/lmcache?ref=dailydoseofds.com)\n**(don’t forget to star it ⭐️)**\nGood day!\nPublished on Jul 7, 2026 \nComments \nShare \nCopy link [ Share to X ](https://x.com/intent/tweet?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Frethinking-kv-caching-for-production-inference%2F&text=Rethinking%20KV%20Caching%20For%20Production%20Inference) [ Share to Facebook ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Frethinking-kv-caching-for-production-inference%2F) [ Share to Linkedin ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Frethinking-kv-caching-for-production-inference%2F)\nCopied \n[ Previous  How to Reduce LLM Costs by 50-60% Using Model Routing  ](https://www.dailydoseofds.com/p/how-to-reduce-llm-costs-by-50-60-using-model-routing/) [ Next  Agentic RL: Environments, Trajectories, and the Training  ](https://www.dailydoseofds.com/rl-course-part-12/)\n[ ![Daily Dose of Data Science](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/12/dailyds-logo-transparent-1.png) ](https://www.dailydoseofds.com)\nA daily column with insights, observations, tutorials and best practices on python and data science. Read by industry professionals at big tech, startups, and engineering students. \n  * Menu \n  * [ Contact ](https://www.dailydoseofds.com/contact/)\n  * [ FAQ ](https://www.dailydoseofds.com/faq/)\n\n\n[Daily Dose of Data Science](https://www.dailydoseofds.com) © 2026 \n[ ](https://x.com/_avichawla \"X \\(Formerly Twitter\\)\")\n",
  "body_base_path": "attachments/dc188d1f1a0f/pages/",
  "body_markdown_path": "attachments/dc188d1f1a0f/pages/archived-body.md",
  "local_html": "attachments/dc188d1f1a0f/pages/archived-body.html",
  "body_source": "prior_capture"
}