{
  "source_file": "0930",
  "source_platform": "Ghost",
  "author": "Avi Chawla / Daily Dose of Data Science",
  "title_listed": "KV Cache Engineering For LLM Serving",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.dailydoseofds.com/p/kv-cache-engineering-for-llm-serving/",
  "publish_date": "2026-09-06",
  "corpus_stage": "",
  "notes": "",
  "id": "1057d60210e8",
  "collected_at": "2026-10-02T13:45:36",
  "title": "KV Cache Engineering for LLM Serving",
  "body_markdown": "Sep 7, 2026\nLLMOps\nKV Cache Engineering for LLM Serving\n12 techniques to manage KV cache in production.\nAvi Chawla\n👉\nLoading an LLM is only the first part of GPU memory planning.\nModel weights stay roughly fixed during inference. The KV cache does not. It grows with every token in the sequence, and by default, every active sequence needs its own cache.\nFor Llama 3.1 70B, one 128K-token sequence requires about 40 GB of BF16 KV cache. Four full-length sequences would add another 160 GB, before accounting for model weights and serving overhead.\nThat creates two separate costs. The cache occupies GPU memory, and the attention layers must repeatedly read it while generating new tokens.\nThe techniques described as KV cache optimizations address different parts of this problem. Some change what the model stores. Some retain fewer tokens or use fewer bits. Others leave the logical cache intact and improve how the serving engine reads, allocates, shares, or moves it.\nThis article organizes twelve techniques by what they reduce, when they can be applied, and what trade-off remains. Each section includes a small runnable example that explains the main idea.\nTo dive deeper into the full LLMOps lifecycle, we have covered every bit of this in the LLMOps course, starting from fundamentals to production.\nStart here →\nThe cache formula\nEach attention layer stores one key and one value for every retained token.\nThe raw cache size for one sequence is:\nThe leading 2 counts keys and values. BF16 uses two bytes per value, while FP8 uses one.\nThat formula also gives us a clean way to group the techniques:\nGQA and MQA reduce the number of KV heads.\nCross-layer attention reduces the number of unique cached layers.\nSliding windows and eviction reduce the number of retained tokens.\nMLA reduces the width of the stored representation.\nQuantization reduces the bytes used by each value.\nHybrid recurrent layers replace a growing cache with fixed-size state.\nPaging and prefix reuse reduce allocation waste and duplicate blocks.\nSparse attention needs a separate line. It may read fewer tokens without removing them from memory, so it can cut attention work while leaving cache capacity unchanged.\nThis distinction matters when the GPU is full. A faster attention kernel doesn’t create room for another sequence unless it also stores fewer bytes.\nTechnique map\nThe visual below puts all 12 techniques in one place, showing what each one changes and whether its main saving comes from fewer heads, layers, tokens, dimensions, bits, or duplicated memory.\nThe table above represents what each technique tries to tackle.\nThe rest of the article works through the full table in order.\n1. GQA and MQA cache fewer heads\nThe cache formula above contains a term for KV heads. Reducing that term is one of the largest architectural savings available.\nInside each transformer layer, every token produces three representations:\nThe query represents what the current token needs.\nThe key describes what each earlier token can match.\nThe value carries the information returned when that match receives attention.\nDuring generation, the current token’s query is compared with cached keys.\nThose comparisons produce attention weights, which combine the cached values. The layer keeps keys and values for later decoding steps. It does not keep the queries.\nAttention splits these representations into smaller parts called heads.\nA query head is one slice of the token’s query representation. It is not the user’s prompt or a separate model request. Different heads can learn to focus on different relationships in the same sequence.\nStandard multi-head attention, or MHA, gives each query head a matching key head and value head. A layer with eight query heads therefore produces eight key heads and eight value heads. Every attention layer performs this projection independently and keeps its own KV cache.\nMulti-query attention, or MQA, keeps all eight query heads but produces only one key head and one value head. Every query head reads from that shared KV pair. The sharing does not cross transformer layers.\nGrouped-query attention, or GQA, sits between these two layouts. It divides the query heads into groups and assigns one KV pair to each group. With eight query heads and two KV heads, four query heads share each KV pair.\nAll three methods operate inside each attention layer. They differ only in how many KV heads that layer stores. Cross-layer attention, covered next, is a separate idea that shares cached information between layers.\nThe GQA paper described it as an intermediate point between MHA and MQA.\nThe authors converted existing MHA checkpoints and used 5 percent of the original pretraining compute for recovery training. Their uptrained GQA models stayed close to MHA quality while approaching MQA speed.\nThe following example uses eight query heads. It shows which KV head each query head reads. It also counts the cached scalars for six tokens:\nquery_heads = 8\ntokens = 6\nhead_size = 8\n\nfor name, kv_heads in [(\"MHA\", 8), (\"GQA\", 2), (\"MQA\", 1)]:\n    queries_per_kv_head = query_heads // kv_heads\n    mapping = [head // queries_per_kv_head for head in range(query_heads)]\n    cached_scalars = 2 * kv_heads * tokens * head_size\n\n    print(name, \"mapping:\", mapping)\n    print(name, \"cached scalars:\", cached_scalars)\n\n# Output:\n\"\"\"\nMHA mapping: [0, 1, 2, 3, 4, 5, 6, 7]\nMHA cached scalars: 768\nGQA mapping: [0, 0, 0, 0, 1, 1, 1, 1]\nGQA cached scalars: 192\nMQA mapping: [0, 0, 0, 0, 0, 0, 0, 0]\nMQA cached scalars: 96\n\"\"\"\nThe mapping lists one KV head number for every query head. MHA uses a different KV head each time. GQA shares KV head 0 across four queries, then does the same with KV head 1. MQA maps every query head to KV head 0.\nThe cache count includes both keys and values. It then multiplies by six tokens and eight scalars per head.\nLlama 3.1 70B uses 64 query heads and 8 KV heads. At 128K context, that produces a 40 GB BF16 cache instead of the 320 GB required by the same shape with 64 KV heads.\nThis code proves the storage ratio. It doesn’t test quality, because head sharing changes what the model learns and must be evaluated after training.\n2. Cross-layer attention\nGQA shares keys and values across heads within one layer. Cross-layer attention shares them across adjacent layers.\nSo a sharing factor of two implies two attention layers use one set of cached keys and values. The model now stores half as many unique layer caches.\nThe Cross-Layer Attention (CLA) paper trained 1B and 3B models and reported another 2x KV cache reduction while keeping accuracy close to the MQA baseline.\nThe next script represents two layers sharing one cache allocation:\ntokens = 32\nkv_heads = 2\nhead_size = 8\n\none_layer_cache = 2 * tokens * kv_heads * head_size\nshared_cache = {\"stored_scalars\": one_layer_cache}\nlayer_caches = [shared_cache, shared_cache]\n\nseparate_total = 2 * one_layer_cache\nshared_total = shared_cache[\"stored_scalars\"]\n\nprint(\"layers point to same cache:\", layer_caches[0] is layer_caches[1])\nprint(\"two separate layer caches:\", separate_total, \"scalars\")\nprint(\"one shared layer cache:\", shared_total, \"scalars\")\n\n# Output:\n\"\"\"\nlayers point to same cache: True\ntwo separate layer caches: 2048 scalars\none shared layer cache: 1024 scalars\n\"\"\"\nOne layer would store 1,024 scalars here. The leading 2 counts keys and values. Two independent layers would therefore store 2,048 scalars.\nBoth items in layer_caches point to the same Python object. The physical storage remains 1,024 scalars even though two logical layers use it.\nCross-layer sharing compounds GQA because it reduces a different term. GQA stores fewer KV heads inside each layer. CLA stores fewer distinct layer caches across the model.\nA standard checkpoint learns separate KV projections for every layer. Each layer expects the cache created by its own weights. Redirecting it to another layer’s cache changes the model’s computation.\nA CLA model trains with the sharing pattern already present. Training lets each dependent layer adapt to the shared cache. The serving engine must then reproduce the same ownership pattern.\nThe checkpoint and engine must agree on which layer owns each cache. This is why CLA cannot be enabled as a generic runtime flag.\n3. Sliding windows\nCLA reduces the number of stored layer caches. Sliding-window attention reduces the tokens stored inside selected layers.\nIn full self-attention, each new token can attend to every earlier token. Each attention layer must therefore keep keys and values for the full sequence. As the sequence grows, every layer’s KV cache grows with it.\nSliding-window attention limits a layer to recent tokens. Suppose the window contains 1,024 positions. That layer stores KV entries for only the latest 1,024 tokens. Each new entry removes the oldest one after the window fills. Its cache then stays fixed at 1,024 positions.\nWe call this a local layer because it sees nearby context. A global layer can still attend to every earlier token. Its cache keeps growing until the sequence ends.\nMany models mix both layer types. Local layers provide most of the memory saving. Occasional global layers preserve access to distant parts of the prompt. The total cache still grows, but much slower than full attention everywhere.\nFor instance, Gemma 3 uses a repeating pattern of five local layers followed by one global layer. Its local window is 1,024 tokens, while the global layer supports the full 128K context:\nThe following example tracks the positions retained by a four-token window:\nwindow_size = 4\nretained_positions = []\n\nfor position in range(10):\n    retained_positions.append(position)\n\n    if len(retained_positions) > window_size:\n        retained_positions.pop(0)\n\n    print(f\"after token {position}: {retained_positions}\")\n\nprint(\"maximum retained positions:\", window_size)\n\n# Output:\n\"\"\"\nafter token 0: [0]\nafter token 1: [0, 1]\nafter token 2: [0, 1, 2]\nafter token 3: [0, 1, 2, 3]\nafter token 4: [1, 2, 3, 4]\nafter token 5: [2, 3, 4, 5]\nafter token 6: [3, 4, 5, 6]\nafter token 7: [4, 5, 6, 7]\nafter token 8: [5, 6, 7, 8]\nafter token 9: [6, 7, 8, 9]\nmaximum retained positions: 4\n\"\"\"\nThe list fills with positions 0 through 3. When position 4 arrives, the list becomes too long. Removing its first item drops position 0.\nThe same step repeats for every later token. Production engines usually use a ring buffer, which overwrites old slots without moving the remaining entries.\nThe first four tokens fill the available slots. Token 4 then removes token 0, so the window shifts to positions 1 through 4. Every later token shifts that range forward by one. After token 9, only positions 6 through 9 remain. The allocation stays fixed at four slots throughout.\nThe ratio also changes with context length. Below 1,024 tokens, both layouts retain the same number of positions. The saving appears only after the sequence exceeds the local window.\n4. Multi-head latent attention\nMulti-head latent attention, or MLA, doesn’t cache a complete key and value for every head. It compresses the hidden state into a smaller latent representation and stores that instead.\nDuring decode, the model uses learned projections to recover the information needed by attention.\nImplementations can absorb parts of those projections into the query and output paths, which avoids reconstructing every full KV tensor in memory.\nDeepSeek-V2 introduced this design. The paper reported a 93.3 percent KV cache reduction compared with DeepSeek 67B and a 5.76x increase in maximum generation throughput:\nAn actual MLA layer uses learned matrices to create and use the latent representation. Those matrix operations hide the basic memory difference in a small example.\nThe following example counts storage for one token. The dimensions are intentionally small. They make the difference between both layouts visible.\nkv_heads = 4\nhead_size = 8\n\nstandard_keys = kv_heads * head_size\nstandard_values = kv_heads * head_size\nstandard_total = standard_keys + standard_values\n\nmla_latent = 8\nmla_rotary_key = 4\nmla_total = mla_latent + mla_rotary_key\n\nprint(\"standard cache:\", standard_keys, \"key +\", standard_values, \"value\")\nprint(\"MLA cache:\", mla_latent, \"latent +\", mla_rotary_key, \"rotary key\")\nprint(\"stored scalars per token:\", standard_total, \"->\", mla_total)\nprint(f\"example reduction: {standard_total / mla_total:.2f}x\")\n\n# Output:\n\"\"\"\nstandard cache: 32 key + 32 value\nMLA cache: 8 latent + 4 rotary key\nstored scalars per token: 64 -> 12\nexample reduction: 5.33x\n\"\"\"\nFour KV heads with eight scalars each produce 32 key scalars. The values require another 32. Standard attention therefore stores 64 scalars per token.\nThe illustrative MLA layout stores eight latent scalars and four positional scalars. Its total is 12. Dividing 64 by 12 gives the example ratio.\nThe 5.33x figure belongs only to this example. Real MLA models choose different latent and positional dimensions.\nMLA is not an inference flag for a standard checkpoint. Its projection weights and cache layout belong to the trained model. Converting another model still requires training.\n5. Replace growing cache with fixed state using Hybrid models\nMamba and recurrent linear-attention layers don’t store one key and value for every old token. They update a fixed-size state as the sequence grows.\nPure recurrent models keep memory nearly constant with context, but full attention remains better at exact retrieval in many settings. Recent models combine both.\nQwen3-Next has 48 layers arranged as 12 repeated blocks. Each block contains three Gated DeltaNet layers and one full-attention layer. Only the 12 full-attention layers create a standard growing KV cache.\nThe model card reports 16 query heads, 2 KV heads, and a head dimension of 256 for those attention layers.\nThe next example compares their memory growth. The attention layer stores one 64-value key and value per token. The recurrent layer keeps one fixed 64-by-64 state.\nwidth = 64\nbytes_per_value = 4\n\nattention_values_per_token = 2 * width\nrecurrent_state_values = width * width\n\nfor tokens in [128, 4096]:\n    attention_kb = tokens * attention_values_per_token * bytes_per_value // 1024\n    recurrent_kb = recurrent_state_values * bytes_per_value // 1024\n\n    print(f\"{tokens} tokens\")\n    print(\"  attention KV:\", attention_kb, \"KB\")\n    print(\"  recurrent state:\", recurrent_kb, \"KB\")\n\n# Output:\n\"\"\"\n128 tokens\n  attention KV: 64 KB\n  recurrent state: 16 KB\n4096 tokens\n  attention KV: 2048 KB\n  recurrent state: 16 KB\n\"\"\"\nThe attention calculation multiplies storage per token by sequence length. The recurrent calculation does not use tokens at all. Its state remains the same size in both cases.\nThe example tracks memory only. It does not implement Mamba or Gated DeltaNet. Those architectures use learned update rules to decide what the fixed state retains.\nThe attention cache grows 32x between the two sequence lengths. The recurrent state remains 16 KB. Hybrid models pay linear cache growth only in their full-attention layers.\nFor the Qwen3-Next attention shape, 12 full-attention layers require 3 GB of BF16 KV at 128K context.\nMaking all 48 layers full attention would raise that growing portion to 12 GB.\nJamba shows the same design choice with a different ratio. It interleaves one attention layer with seven Mamba layers. At 256K context, the Jamba paper reports a 4 GB KV cache, compared with 32 GB for Mixtral.\nThis layout belongs to the trained model. A serving engine cannot replace arbitrary attention layers with recurrent ones at runtime.\n6. Compressed sparse attention\nFull attention does two expensive things. It stores a separate KV entry for every earlier token, then reads every stored entry when generating the next token.\nSparse attention changes only the second part.\nIt reads fewer entries for each new token, but it may still keep the complete cache in memory. That can make generation faster without making the resident KV cache smaller.\nCompressed sparse attention focuses on optimizing both parts.\nFirst, it combines several neighbouring token entries into one compressed entry. It stores that shorter sequence instead of one long-range entry per token.\nThen, for each new token, a selector chooses only the compressed entries that appear most relevant. Attention reads that selected subset.\nDeepSeek V4 uses this design.\nIf the compression factor is m, every m token entries become one stored entry. So a context of n tokens produces roughly n / m compressed entries.\nThe model then selects k of those entries for attention. A small sliding window preserves finer detail for the most recent tokens.\nAt one million tokens, DeepSeek reports that V4-Pro uses 10 percent of the KV cache and 27 percent of the single-token inference FLOPs required by DeepSeek V3.2. V4-Flash goes further at 7 percent of the cache and 10 percent of the FLOPs\nThese are whole-model results. They include V4’s hybrid attention layout and precision choices, so they should not be read as the isolated effect of compression alone.\nThe next block isolates the two reductions. Compressing 32 long-range entries in groups of four leaves eight stored entries. If attention selects two of those eight, it reads only two compressed entries for the current step.\nfull_token_entries = 32\ntokens_per_group = 4\nselected_entries = 2\n\ncompressed_entries = full_token_entries // tokens_per_group\n\nprint(\"full token entries:\", full_token_entries)\nprint(\"compressed entries stored:\", compressed_entries)\nprint(\"compressed entries read:\", selected_entries)\nprint(\"storage reduction:\", f\"{full_token_entries / compressed_entries:.0f}x\")\n\n# Output:\n\"\"\"\nfull token entries: 32\ncompressed entries stored: 8\ncompressed entries read: 2\nstorage reduction: 4x\n\"\"\"\nDividing 32 tokens into groups of four creates eight compressed entries. That is the storage reduction. Selecting two of those eight entries is the separate attention-work reduction.\nThe cache now holds eight long-range entries instead of 32. Attention reads two of them. A small local window separately preserves recent token detail.\nThis technique has to be built into the model and training process. A serving engine cannot convert an existing full-attention model into compressed sparse attention by changing a cache setting.\n7. Query-aware sparse reads\nThe previous technique reduced both storage and attention work.\nSometimes, changing the model architecture is not an option. We may still want to reduce how much cache each decode step reads.\nRecall what happens while the model generates a token. The new token produces a query, which acts like a search signal. Full attention compares it with every stored key. A long prompt therefore forces the GPU to load a large cache repeatedly.\nQuest is a technique that reduces those reads without deleting any KV entries.\nIt divides the cache into pages.\nA page is a small group of neighbouring token entries.\nEach page also stores the minimum and maximum key values found inside it.\nFor every new query, Quest scores these short page summaries first. It then loads only the pages with the highest scores. The full cache remains stored because a page ignored now may matter later.\nThe Quest paper reports up to 7.03x lower self-attention latency. Its complete inference system reached up to 2.23x speedup.\nThe example below uses four pages. Each page contains two keys. A key has only two numbers here, which keeps the scoring visible.\nimport numpy as np\n\npages = np.array([\n    [[1, 0], [2, 1]],\n    [[5, 1], [4, 2]],\n    [[0, 6], [1, 5]],\n    [[-2, -1], [-1, 0]],\n])\nquery = np.array([1.0, 0.5])\n\npage_min = pages.min(axis=1)\npage_max = pages.max(axis=1)\npage_scores = np.maximum(query * page_min, query * page_max).sum(axis=1)\nselected_pages = np.argsort(page_scores)[-2:]\n\nfor page, score in enumerate(page_scores):\n    print(f\"page {page} score: {score:.1f}\")\n\nprint(\"selected pages:\", np.sort(selected_pages).tolist())\nprint(\"KV entries read:\", len(selected_pages) * 2, \"of\", pages.size // 2)\nprint(\"KV entries still stored:\", pages.size // 2)\n\n# Output:\n\"\"\"\npage 0 score: 2.5\npage 1 score: 6.0\npage 2 score: 4.0\npage 3 score: -1.0\nselected pages: [1, 2]\nKV entries read: 4 of 8\nKV entries still stored: 8\n\"\"\"\nThe pages array contains eight keys in total. The next two lines create one minimum and one maximum summary per page. Quest uses similar summaries to estimate a page’s best possible attention score.\nThe query prefers large values in both key dimensions. Pages 1 and 2 receive the highest scores, so the example reads their four keys. It leaves the other four keys untouched during this step.\nIf you notice the last line, the step reads four entries, but all eight remain resident. Quest reduces cache traffic and attention work. It does not reduce the memory needed to hold the full cache.\nIn practice, Quest uses many-dimensional keys and dedicated GPU code. It also repeats the selection for every query.\n8. Quantization\nQuest keeps every KV entry and skips some reads. Quantization takes another route. It keeps every entry but stores each number with fewer bits.\nKeys and values normally contain floating-point numbers. BF16 stores each number with 16 bits. FP8 uses 8 bits, so its raw payload is roughly half as large. A 4-bit format cuts the payload to 25%.\nQuantization maps many original numbers onto a smaller set of allowed values. A scale controls the gap between those values. The engine stores the small integer code and enough scale information to interpret it later.\nThe choice of scale affects accuracy. One extreme value can make a shared scale too wide. Most ordinary values then get rounded too aggressively.\nKIVI handles keys and values differently. It groups key values by channel, which means the same feature across tokens. It groups value-cache numbers by token. The KIVI paper reports 2.6x lower peak memory and up to 4x larger batches with its 2-bit cache.\nThe small example below shows the basic mapping. It converts six BF16-style values into signed 4-bit codes. It then converts those codes back into approximate values.\nimport numpy as np\n\nvalues = np.array([-1.0, -0.55, -0.10, 0.25, 0.60, 1.0])\nbits = 4\nlargest_code = 2 ** (bits - 1) - 1\nscale = np.max(np.abs(values)) / largest_code\n\ncodes = np.rint(values / scale)\ncodes = np.clip(codes, -largest_code, largest_code).astype(np.int8)\nrestored = codes * scale\n\nprint(\"original values:\", values.tolist())\nprint(\"4-bit codes:\", codes.tolist())\nprint(\"restored values:\", np.round(restored, 3).tolist())\nprint(\"raw payload:\", len(values) * 16, \"bits ->\", len(values) * 4, \"bits\")\n\n# Output:\n\"\"\"\noriginal values: [-1.0, -0.55, -0.1, 0.25, 0.6, 1.0]\n4-bit codes: [-7, -4, -1, 2, 4, 7]\nrestored values: [-1.0, -0.571, -0.143, 0.286, 0.571, 1.0]\nraw payload: 96 bits -> 24 bits\n\"\"\"\nSigned 4-bit storage provides codes from -7 through 7 in this example. Dividing by scale maps each original value to a code. Multiplying by the same scale produces the approximate value used during attention.\nAs you may have noticed, the restored values are close, but not identical. That rounding difference is quantization error. The final line counts only the six stored values. A real cache also stores scale metadata.\nThe example uses one scale for all six values. KIVI uses smaller groups because each group can fit its own numerical range. This usually reduces rounding error, though it requires more scale metadata.\nIf you use vLLM, it exposes FP8 cache storage directly:\nMODEL_ID=Qwen/Qwen2.5-7B-Instruct\nvllm serve \"$MODEL_ID\" --kv-cache-dtype fp8\nThe first line defines the model name. The second starts a vLLM server and asks it to store the KV cache in FP8. This command keeps running because it launches the inference server.\nRecent vLLM versions can also leave selected layers in their native type. This is useful when a sliding-window layer proves more sensitive:\nMODEL_ID=google/gemma-3-4b-it\nvllm serve \"$MODEL_ID\" \\\n  --kv-cache-dtype fp8 \\\n  --kv-cache-dtype-skip-layers sliding_window\nThis version leaves sliding-window layers in their original format. It quantizes the remaining cache layers. The option helps when those local-attention layers lose too much accuracy under FP8.\nQuantization preserves every token position. The next technique reduces memory by removing positions instead.\n9. Eviction\nQuantization keeps all token positions. That may still be too large for a long context. Eviction sets a fixed position budget and discards entries outside it.\nThe difficult question is which positions should not be evicted. Recent tokens often matter because they contain the current conversation. Older tokens may still carry instructions, facts, or tool results needed later.\nH2O is a technique that keeps recent positions and older heavy hitters. A heavy hitter is a token that has accumulated high attention during earlier steps. Once H2O evicts another token, its KV entry is no longer available.\nOther methods estimate importance differently. SnapKV studies attention near the end of the input prompt. It uses that observation window to choose prompt positions before generation.\nPyramidKV gives lower layers larger budgets and higher layers smaller ones, based on the attention patterns reported by its authors.\nSnapKV reports 8.2x higher memory efficiency and 3.6x faster generation with 16K-token inputs. PyramidKV reports matching full-cache performance while retaining 12 percent of the cache in its LongBench experiments.\nThe following example takes a snapshot after ten tokens. Their scores represent attention accumulated during earlier decode steps. The cache can retain only six positions, including the latest two.\nscores = [9.0, 0.2, 4.0, 0.1, 0.3, 6.0, 0.2, 0.4, 0.1, 0.2]\nbudget = 6\nrecent_count = 2\n\npositions = list(range(len(scores)))\nrecent = positions[-recent_count:]\nolder = positions[:-recent_count]\n\nheavy_hitter_count = budget - recent_count\nheavy_hitters = sorted(\n    older,\n    key=lambda position: scores[position],\n    reverse=True,\n)[:heavy_hitter_count]\n\nretained = sorted(heavy_hitters + recent)\n\nprint(\"heavy hitters:\", sorted(heavy_hitters))\nprint(\"recent positions:\", recent)\nprint(\"retained positions:\", retained)\nprint(\"cache entries:\", len(retained), \"of\", len(scores))\n\n# Output:\n\"\"\"\nheavy hitters: [0, 2, 5, 7]\nrecent positions: [8, 9]\nretained positions: [0, 2, 5, 7, 8, 9]\ncache entries: 6 of 10\n\"\"\"\nPositions 8 and 9 are retained because they are the newest. Four slots remain. The sort selects positions 0, 2, 5, and 7 because they have the largest scores among older tokens.\nThe union of those two groups produces the final six-position cache. Positions 1, 3, 4, and 6 are discarded.\nReal H2O updates its heavy-hitter statistics as generation proceeds. This smaller example starts with the accumulated scores so the selection rule remains visible.\nThat said, eviction needs a cautious evaluation. Importance changes across turns. A tool result that receives little attention now may become necessary after several more calls. System instructions and delimiters can also be disproportionately important.\nEviction evaluations should therefore use complete agent traces and delayed references. They should also include structured prompts. A good Needle-in-a-Haystack score does not cover those cases.\n10. Paging\nSo far, we have looked at what one request stores or reads. Paging solves a different problem: how the serving engine allocates cache memory across many requests.\nA simple allocator may reserve one continuous memory region per request. It often reserves enough space for the request’s maximum length. Shorter requests leave part of that region unused. Free space also gets split into gaps as requests finish.\nPagedAttention divides GPU cache memory into equal-sized blocks. Each block holds KV entries for a fixed number of tokens. A request can use blocks found anywhere in the shared pool, so its blocks do not need to lie next to each other.\nWhen a request finishes, the allocator returns its blocks to the pool. Another request can reuse them immediately. The PagedAttention paper reports near-zero KV cache waste and 2x to 4x higher throughput than the systems tested.\nA fixed pool can hide the effect of quantization. The engine may reserve the same 8 GB before and after enabling FP8. Smaller entries increase the number of tokens that fit inside that pool. The allocated memory shown by nvidia-smi may remain unchanged.\nThe example below begins with six free blocks. Request A takes three. Request B takes two. A then finishes and returns its blocks before request C arrives.\nfree_blocks = [0, 1, 2, 3, 4, 5]\n\nrequest_a = free_blocks[:3]\nfree_blocks = free_blocks[3:]\n\nrequest_b = free_blocks[:2]\nfree_blocks = free_blocks[2:]\n\nfree_blocks.extend(request_a)\nfree_after_a = free_blocks.copy()\nrequest_c = free_blocks[:2]\nfree_blocks = free_blocks[2:]\n\nprint(\"request A blocks:\", request_a)\nprint(\"request B blocks:\", request_b)\nprint(\"free after A finishes:\", free_after_a)\nprint(\"request C blocks:\", request_c)\n\n# Output:\n\"\"\"\nrequest A blocks: [0, 1, 2]\nrequest B blocks: [3, 4]\nfree after A finishes: [5, 0, 1, 2]\nrequest C blocks: [5, 0]\n\"\"\"\nAfter A finishes, the free pool contains block 5 and A’s returned blocks. Request C takes blocks 5 and 0. Those block numbers are not adjacent, but the block table records their order for C.\nThis is the central paging idea. The allocator reuses available blocks instead of searching for one large continuous region.\nvLLM also tracks which request owns each block. Its attention code can read the non-adjacent blocks in logical sequence order. The short example isolates allocation and reuse.\nQuantization still changes the capacity inside this pool. An 8 GB pool holds 26,214 tokens at 320 KB per token, compared with 52,428 tokens at 160 KB. Total allocated GPU memory remains 8 GB in both cases.\n11. Prefix reuse\nPaging reuses blocks after a request finishes. Prefix caching can share blocks while several requests are still active. It applies when those requests begin with exactly the same tokens.\nThis happens often with system prompts and tool definitions. Without prefix caching, the engine computes and stores another KV copy for every request. The repeated prefix consumes memory and repeats the prefill work that created it.\nAutomatic prefix caching stores each completed prefix block under a lookup key. When another request has the same token prefix, it points to the existing KV block. New computation begins only after the requests diverge.\nvLLM builds each lookup key from the current block and its preceding blocks. It also includes details such as adapter identifiers when they affect compatibility. The vLLM prefix-caching design documents this structure.\nThe example below uses four-token blocks. Each lookup key contains the entire prefix up to that block. This is a readable stand-in for vLLM’s chained hashes.\ndef prefix_keys(tokens, block_size):\n    block_ends = range(block_size, len(tokens) + 1, block_size)\n    return [tuple(tokens[:end]) for end in block_ends]\n\nfirst = [10, 11, 12, 13, 20, 21, 22, 23, 30, 31, 32, 33]\nsecond = [10, 11, 12, 13, 20, 21, 22, 23, 40, 41, 42, 43]\n\ncache = set()\nfirst_keys = prefix_keys(first, block_size=4)\nfirst_hits = sum(key in cache for key in first_keys)\ncache.update(first_keys)\n\nsecond_keys = prefix_keys(second, block_size=4)\nsecond_hits = sum(key in cache for key in second_keys)\n\nprint(\"first request reused:\", first_hits, \"of\", len(first_keys), \"blocks\")\nprint(\"second request reused:\", second_hits, \"of\", len(second_keys), \"blocks\")\nprint(\"second request computes:\", len(second_keys) - second_hits, \"new block\")\n\n# Output:\n\"\"\"\nfirst request reused: 0 of 3 blocks\nsecond request reused: 2 of 3 blocks\nsecond request computes: 1 new block\n\"\"\"\nThe first request has no cache entries to reuse. It adds keys for its first four, eight, and twelve tokens. The second request has the same first eight tokens, so its first two keys match.\nThe final four tokens differ. That changes the last prefix key, so the second request computes one new block.\nUsing the full prefix prevents a false match. Two requests cannot reuse a block merely because four tokens match somewhere in the middle. Their preceding context must match as well.\nPrefix reuse depends on exact tokenized prefixes. A timestamp in the system prompt, reordered JSON keys in a tool schema, or different chat-template whitespace can break the match.\nStable prompt templates therefore improve the hit rate. The model and cache format can remain unchanged while duplicate storage drops.\n12. Cache Offloading from GPU\nPrefix reuse helps when another request begins with tokens the engine has already processed. Offloading handles a different problem. It moves KV blocks that are not currently needed from GPU memory to a larger but slower memory tier, usually CPU memory.\nThose blocks may belong to a sequence paused by the scheduler or to a cached prefix retained for reuse. When the engine needs them again, it copies them back to the GPU.\nThere is one important limitation. A new OpenAI-compatible API request does not automatically resume the KV cache from an earlier request. Reuse happens only if the engine still tracks the sequence, recognizes an exact prefix-cache match, or restores the cache through an external storage system.\nOffloading frees GPU memory without deleting the stored KV data. It does not reduce the total number of bytes held across GPU and CPU memory. The trade-off appears when the engine must transfer those blocks back before computation can continue.\nThe example below gives the GPU room for two sessions. Adding session C moves the oldest session, A, to CPU memory. Resuming A brings it back and moves B out.\ngpu_cache = [\"cache-a\", \"cache-b\"]\ncpu_cache = []\n\n# Cache C needs GPU space, so cache A moves to CPU memory.\ncpu_cache.append(gpu_cache.pop(0))\ngpu_cache.append(\"cache-c\")\n\nprint(\"GPU after cache C arrives:\", gpu_cache)\nprint(\"CPU after cache C arrives:\", cpu_cache)\n\n# Cache A is needed again, so it returns to the GPU.\ncpu_cache.remove(\"cache-a\")\ncpu_cache.append(gpu_cache.pop(0))\ngpu_cache.append(\"cache-a\")\n\nprint(\"GPU after cache A returns:\", gpu_cache)\nprint(\"CPU after cache A returns:\", cpu_cache)\n\n# Output:\n\"\"\"\nGPU after cache C arrives: ['cache-b', 'cache-c']\nCPU after cache C arrives: ['cache-a']\nGPU after cache A returns: ['cache-c', 'cache-a']\nCPU after cache A returns: ['cache-b']\n\"\"\"\nThe first move leaves caches B and C on the GPU while cache A remains available in CPU memory. When cache A is needed again, the example brings it back and moves cache B out.\nThe lists contain cache names instead of tensors. A real engine performs this operation on KV blocks and tracks which request or reusable prefix owns each block.\nThe example assumes that the engine has retained enough information to identify cache A later. Sending an unrelated API request with the same label would not restore it automatically.\nvLLM can enable a 16 GB CPU offloading buffer directly:\nMODEL_ID=Qwen/Qwen2.5-7B-Instruct\n\nvllm serve \"$MODEL_ID\" \\\n  --kv-offloading-size 16 \\\n  --kv-offloading-backend native\nThe first option sets the CPU buffer size in GB. The second selects vLLM’s native CPU backend. Like the earlier serving command, this process keeps running after startup.\nOffloading trades GPU capacity for transfer time. Measure resume latency and CPU bandwidth before enabling a large offload tier.\nHow to use them together\nSome of these methods compose because they reduce different terms. Architecture-level methods still depend on how the checkpoint was trained.\nFor instance, take the 40 GB GQA cache from the first example.\nCLA2 can cut the unique layers in half, producing 20 GB.\nFP8 can halve the bytes per value, producing roughly 10 GB.\nA 50 percent token budget would reduce the resident cache to roughly 5 GB.\nThat multiplication is useful for capacity planning, but it hides model quality and kernel support.\nCLA requires training.\nFP8 introduces quantization error.\nEviction discards context.\nAnd the serving engine must also support the resulting layout.\nThese are some situations you may face, along with solutions:\nWhen choosing a model → Inspect KV head count, local and global layer ratios, latent-attention dimensions, and recurrent layers. These determine the starting cache before deployment.\nWhen serving an existing model → Test FP8 first, then standardize prefixes. Both preserve the token set. Add eviction only after workload-specific quality tests.\nWhen memory appears unchanged → Inspect cache capacity and block counts. A fixed pool can absorb the savings while total allocated GPU memory stays flat.\nWhen latency is the problem → Measure cache reads and attention time. Quest-style sparse reads may help even when they don’t free capacity.\nWhen inactive sessions occupy the GPU → Test offloading and scheduling. Moving cold blocks can matter more than compressing the active ones further.\nKV cache reduction is easier to reason about once each technique has a specific target.\nThis table summarizes all of the techniques we cover, what they alter, the takeaway from it, and what part they don’t handle:\nStart with the constraint you need to change, then pick the technique that handles it.\nTo dive deeper into the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:\nRead Part 1 on fundamentals of LLMOps here →\nRead Part 2 on understanding the core building blocks of LLMs →\nRead Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →\nRead Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →\nRead Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →\nRead Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →\nRead Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →\nRead Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →\nRead Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →\nRead Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →\nRead Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →\nRead Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →\nRead Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →\nRead Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.\n👉 Over to you: how do you reduce KV cache in production?\nGood day!\nPublished on Sep 7, 2026\nComments\nShare\nCopy link\nShare to X\nShare to Facebook\nShare to Linkedin\nCopied\nPrevious\nPreloading Knowledge Into a Model Instead of Retrieving It (Part C)\nNext\nFirst Steps Towards a Production-Ready Agent with LangChain",
  "metadata": {
    "title": "KV Cache Engineering for LLM Serving",
    "description": "12 techniques to manage KV cache in production.",
    "keywords": null,
    "author": null,
    "og:site_name": "Daily Dose of Data Science",
    "og:type": "article",
    "og:title": "KV Cache Engineering for LLM Serving",
    "og:description": "12 techniques to manage KV cache in production.",
    "og:url": "https://www.dailydoseofds.com/p/kv-cache-engineering-for-llm-serving/",
    "og:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w1200/2026/09/thumbnail-kv-cache-reduction.png",
    "og:image:width": "1200",
    "og:image:height": "480",
    "twitter:card": "summary_large_image",
    "twitter:title": "KV Cache Engineering for LLM Serving",
    "twitter:description": "12 techniques to manage KV cache in production.",
    "twitter:url": "https://www.dailydoseofds.com/p/kv-cache-engineering-for-llm-serving/",
    "twitter:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w1200/2026/09/thumbnail-kv-cache-reduction.png",
    "twitter:label1": "Written by",
    "twitter:data1": "Avi Chawla",
    "twitter:label2": "Filed under",
    "twitter:data2": "LLMOps",
    "twitter:site": "@_avichawla",
    "article:published_time": "2026-09-06T20:29:35.000Z",
    "article:modified_time": "2026-09-06T20:29:35.000Z",
    "article:tag": "LLMOps"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/1057d60210e8/code/body-000.txt",
      "chars": 574,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-001.txt",
      "chars": 594,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-002.txt",
      "chars": 610,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-003.txt",
      "chars": 656,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-004.txt",
      "chars": 556,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-005.txt",
      "chars": 483,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-006.txt",
      "chars": 779,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-007.txt",
      "chars": 706,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-008.txt",
      "chars": 77,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-009.txt",
      "chars": 125,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-010.txt",
      "chars": 714,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-011.txt",
      "chars": 567,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-012.txt",
      "chars": 878,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-013.txt",
      "chars": 704,
      "language": "text"
    },
    {
      "file": "attachments/1057d60210e8/code/body-014.txt",
      "chars": 120,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "bytes": 11344,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/000.jpg",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "alt": "Avi Chawla",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!AnkW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4105492a-1bf1-4f61-9fbb-722e88269910_680x456.png",
      "bytes": 121002,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/001.png",
      "src": "https://substackcdn.com/image/fetch/$s_!AnkW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4105492a-1bf1-4f61-9fbb-722e88269910_680x456.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Azxq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f22919f-8c84-4bbe-b6b1-7afdc001c215_680x380.png",
      "bytes": 79699,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/002.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Azxq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f22919f-8c84-4bbe-b6b1-7afdc001c215_680x380.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!S_5O!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20ed54e6-a2c7-43b3-be37-82553bf14016_1040x903.png",
      "bytes": 145156,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/003.png",
      "src": "https://substackcdn.com/image/fetch/$s_!S_5O!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20ed54e6-a2c7-43b3-be37-82553bf14016_1040x903.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!luhm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c7ba56a-f213-4b81-948d-1da66f7bd9c3_680x380.png",
      "bytes": 71390,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/004.png",
      "src": "https://substackcdn.com/image/fetch/$s_!luhm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c7ba56a-f213-4b81-948d-1da66f7bd9c3_680x380.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!hoad!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf85db60-722e-40a2-bf8e-7fa695b51ff8_680x380.png",
      "bytes": 126155,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/005.png",
      "src": "https://substackcdn.com/image/fetch/$s_!hoad!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf85db60-722e-40a2-bf8e-7fa695b51ff8_680x380.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!TM03!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e2616ba-cc40-40a5-b792-352b5e11ccc0_680x380.png",
      "bytes": 95120,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/006.png",
      "src": "https://substackcdn.com/image/fetch/$s_!TM03!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e2616ba-cc40-40a5-b792-352b5e11ccc0_680x380.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!otgq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb67527b8-e5b5-444d-af90-71cb144647e0_680x456.png",
      "bytes": 114790,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/007.png",
      "src": "https://substackcdn.com/image/fetch/$s_!otgq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb67527b8-e5b5-444d-af90-71cb144647e0_680x456.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Y6qB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cd4f57d-5ab5-4759-af5c-f5a44a082d05_680x380.png",
      "bytes": 93962,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/008.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Y6qB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cd4f57d-5ab5-4759-af5c-f5a44a082d05_680x380.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!E5nz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F985d1321-72af-422c-a5a1-997a425cbfc9_680x380.png",
      "bytes": 108673,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/009.png",
      "src": "https://substackcdn.com/image/fetch/$s_!E5nz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F985d1321-72af-422c-a5a1-997a425cbfc9_680x380.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!0IOL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9aa1eb83-2288-4c17-9b52-b62c21920ce0_680x508.png",
      "bytes": 115681,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/010.png",
      "src": "https://substackcdn.com/image/fetch/$s_!0IOL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9aa1eb83-2288-4c17-9b52-b62c21920ce0_680x508.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!7kEX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59021c73-1f7f-408a-9979-d9b2112fd8ce_680x638.png",
      "bytes": 91581,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/011.png",
      "src": "https://substackcdn.com/image/fetch/$s_!7kEX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59021c73-1f7f-408a-9979-d9b2112fd8ce_680x638.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!EQ_Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F11598f7c-2575-42d2-ac4a-45e6401d4ab3_680x380.png",
      "bytes": 94945,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/012.png",
      "src": "https://substackcdn.com/image/fetch/$s_!EQ_Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F11598f7c-2575-42d2-ac4a-45e6401d4ab3_680x380.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!U3p0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F794ce750-16de-4c57-b914-bb4735a37751_680x380.png",
      "bytes": 94856,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/013.png",
      "src": "https://substackcdn.com/image/fetch/$s_!U3p0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F794ce750-16de-4c57-b914-bb4735a37751_680x380.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!NU-3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46709c5b-d6aa-4b1a-bb72-057d911a488f_680x380.png",
      "bytes": 129046,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/014.png",
      "src": "https://substackcdn.com/image/fetch/$s_!NU-3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46709c5b-d6aa-4b1a-bb72-057d911a488f_680x380.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!tgig!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3559bbb-48fd-4a88-8d5a-5ebb0330d307_679x379.png",
      "bytes": 114527,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/015.png",
      "src": "https://substackcdn.com/image/fetch/$s_!tgig!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3559bbb-48fd-4a88-8d5a-5ebb0330d307_679x379.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!-bOS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F184dea06-05b6-4a8c-b005-bc194d7a4743_679x379.png",
      "bytes": 97022,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/016.png",
      "src": "https://substackcdn.com/image/fetch/$s_!-bOS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F184dea06-05b6-4a8c-b005-bc194d7a4743_679x379.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!sYAe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc1be590c-234c-4315-b770-e3848a3d0630_679x379.png",
      "bytes": 103852,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/017.png",
      "src": "https://substackcdn.com/image/fetch/$s_!sYAe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc1be590c-234c-4315-b770-e3848a3d0630_679x379.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!OQU6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89b90c0a-aafa-45c6-8881-f3f0d91d214f_680x380.png",
      "bytes": 102842,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/018.png",
      "src": "https://substackcdn.com/image/fetch/$s_!OQU6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89b90c0a-aafa-45c6-8881-f3f0d91d214f_680x380.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Ywq5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccac114e-4a76-4a81-8b48-6ff33dd757d6_680x371.png",
      "bytes": 121874,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/019.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Ywq5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccac114e-4a76-4a81-8b48-6ff33dd757d6_680x371.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!pvSE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b01824d-c48f-45be-92da-cc0635446607_680x380.png",
      "bytes": 104320,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/020.png",
      "src": "https://substackcdn.com/image/fetch/$s_!pvSE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b01824d-c48f-45be-92da-cc0635446607_680x380.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!hXwX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff319feb2-af2d-4f31-9800-f29b491f0319_680x380.png",
      "bytes": 93932,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/021.png",
      "src": "https://substackcdn.com/image/fetch/$s_!hXwX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff319feb2-af2d-4f31-9800-f29b491f0319_680x380.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!LT8_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F772bf0d0-ea2b-429c-be30-5dc138f921e6_680x456.png",
      "bytes": 108352,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/022.png",
      "src": "https://substackcdn.com/image/fetch/$s_!LT8_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F772bf0d0-ea2b-429c-be30-5dc138f921e6_680x456.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!o6Qt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac757cd9-9de0-40fe-85ca-5a8781ed2961_680x456.png",
      "bytes": 102652,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/023.png",
      "src": "https://substackcdn.com/image/fetch/$s_!o6Qt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac757cd9-9de0-40fe-85ca-5a8781ed2961_680x456.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!lHep!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c670eaf-3b1d-4ef7-8e5c-b092d531a171_1040x903.png",
      "bytes": 145156,
      "reused": true,
      "local_path": "attachments/1057d60210e8/images/024.png",
      "src": "https://substackcdn.com/image/fetch/$s_!lHep!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c670eaf-3b1d-4ef7-8e5c-b092d531a171_1040x903.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-1/",
      "label": "Start here →",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-2",
      "label": "Read Part 2 on understanding the core building blocks of LLMs →",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-3",
      "label": "Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixt",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-4",
      "label": "Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applic",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-5",
      "label": "Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and differ",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-6",
      "label": "Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-7",
      "label": "Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techn",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-8",
      "label": "Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic contex",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-9",
      "label": "Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong und",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-10",
      "label": "Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for ev",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-11",
      "label": "Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-12/",
      "label": "Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF,",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-13/",
      "label": "Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model ",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/llmops-crash-course-part-14",
      "label": "Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decision",
      "type": "a"
    },
    {
      "href": "https://x.com/intent/tweet?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fkv-cache-engineering-for-llm-serving%2F&text=KV%20Cache%20Engineering%20for%20LLM%20Serving",
      "label": "Share to X",
      "type": "a"
    },
    {
      "href": "https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fkv-cache-engineering-for-llm-serving%2F",
      "label": "Share to Facebook",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fkv-cache-engineering-for-llm-serving%2F",
      "label": "Share to Linkedin",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/building-rag-systems-course-part-15-with-implementation/",
      "label": "Previous Preloading Knowledge Into a Model Instead of Retrieving It (Part C)",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/ai-agents-with-langgraph-course-part-1-with-implementation/",
      "label": "Next First Steps Towards a Production-Ready Agent with LangChain",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:54Z",
  "source_url": "https://www.dailydoseofds.com/p/kv-cache-engineering-for-llm-serving/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 39472,
  "expected_body_images": 25,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/1057d60210e8/pages/source.html",
  "local_html": "attachments/1057d60210e8/pages/article.html",
  "body_base_path": "attachments/1057d60210e8/pages/",
  "body_markdown_path": "attachments/1057d60210e8/pages/article.md"
}