{
  "source_file": "0930",
  "source_platform": "Ghost",
  "author": "Avi Chawla / Daily Dose of Data Science",
  "title_listed": "KV vs Prefix vs Prompt vs Semantic Caching",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.dailydoseofds.com/p/kv-vs-prefix-vs-prompt-vs-semantic-caching/",
  "publish_date": "2026-08-30",
  "corpus_stage": "",
  "notes": "",
  "id": "9c3f9ccc7fe7",
  "collected_at": "2026-10-02T13:45:46",
  "title": "KV vs Prefix vs Prompt vs Semantic Caching",
  "body_markdown": "Aug 28, 2026\nLLMOps\nKV vs Prefix vs Prompt vs Semantic Caching\n...explained with best practices in production.\nAvi Chawla\n👉\nFour things in an LLM stack store four different objects, and all of them get called caching.\nThe KV cache stores attention tensors for one request.\nPrefix caching stores those same tensors on the server, keyed by a hash chain over token IDs.\nPrompt caching is the provider’s billed version of that same lookup, at 0.1x the base input rate on a read against a 1.25x premium on the write.\nA semantic cache stores finished response strings, keyed by cosine similarity over an embedding.\nThe first three are exact-match and correctness-neutral, so a miss costs you money and latency. The fourth is fuzzy-match, and it will hand you a wrong answer with a 200.\nSo today, let’s go through all four, what each one stores, and what quietly breaks it.\nEverything here runs on one machine, CPU included, with a 360M parameter model. There is also one Anthropic API example and one small semantic cache built on sentence-transformers. Where a mechanism only exists inside a serving engine, we walk the logic in pseudocode instead of pretending it is reproducible on a laptop.\nAlso, the cache API changed shape in transformers v5, so the snippets below assume v5 or later. On v4, the equivalents are DynamicCache() with no config argument and torch_dtype= instead of dtype=.\npip install \"transformers>=5.0\" torch\n\n# only for the quantized cache example\npip install optimum-quanto\n\n# only for the semantic cache example\npip install sentence-transformers \n\n# only for the prompt caching example\npip install anthropic\n1) The KV cache\nDuring prefill, the model computes a key and value vector for every prompt token at every layer and stores them.\nDecoding then attends over those stored vectors and appends one new pair per generated token, instead of recomputing the whole sequence each step.\nQueries don’t get cached, and the reason is causal masking. A token’s query vector is used once, at the step that token is processed, and never read again. Its key and value are read by every token that comes after it, so those are the two most important vectors to save.\nWithout storing them, each decode step requires a matrix-matrix multiply over the full sequence that has been generated so far.\nWith it, the step becomes a matrix-vector multiply over one new token, which is far fewer FLOPs.\nThe video below depicts LLM inference with and without KV caching:\n0:00\n/\n0:46\n1×\nWhile this reduces the computation on each token, you have to load the entire cache from HBM on every single step, so decode is no longer compute-bound but rather becomes memory bandwidth-bound.\nAttention kernels finish faster than the cache can be streamed in, and the GPU spends most of a decode step waiting on memory.\nKV cache growth with each token\nThe\ntransformers\nlibrary exposes the cache as a first-class object, so you can hold it, inspect it, and pass it back in.\nHere is a minimal code demo of it:\nimport torch\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache\n\nmodel_id = \"HuggingFaceTB/SmolLM2-360M-Instruct\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_id, dtype=torch.bfloat16, device_map=\"auto\"\n)\n\ninputs = tokenizer(\"The capital of France is\", return_tensors=\"pt\")\ninputs = inputs.to(model.device)\n\npast_key_values = DynamicCache(config=model.config)\n\nout = model.generate(\n    **inputs,\n    do_sample=False,\n    max_new_tokens=20,\n    past_key_values=past_key_values,\n)\n\n>>> print(tokenizer.decode(out[0], skip_special_tokens=True))\n\"\"\"The capital of France is Paris. It is the largest city in\nFrance and the second-largest city in the European Union.\"\"\"\n\n>>> print(\"prompt tokens: \", inputs[\"input_ids\"].shape[1])\n\"prompt tokens: 5\"\n\n>>> print(\"total tokens: \", out.shape[1])\n\"total tokens:  25\"\n\n>>> print(\"cache length: \", past_key_values.get_seq_length())\n\"cache length:  24\"\nNormally, you invoke the\ngenerate\nmethod and the cache is created and destroyed internally, invisible to you. Here we construct a DynamicCache ourselves and hand it in, which means we still hold a reference to it after generation finishes.\nget_seq_length()\nthen reports how many token positions the cache holds. When you run this, the output contains the prompt length plus the tokens generated, minus one.\nThe final token's key and value are computed but never attended over by anything.\nThis code shows the cache holds one entry per token seen, and it grows by exactly one entry per decode step.\nDynamicCache is used as the default because it grows as generation proceeds rather than pre-allocating, so short requests don't reserve memory they will never use.\nThe cache decides how many requests can fit on a GPU. Its size is fixed by the model shape and grows linearly with token count, since every layer holds a key and value tensor for every KV head.\nFor a 70B model at BF16, a single 128K context holds around 40 GB of cache, comparable to the entire model at 4-bit weights.\nThese are some ways to reduce this. For instance, Grouped-query attention shares one key and value head across a group of query heads, which shrinks the cache and raises FLOPs per byte of data loaded.\nMulti-head latent attention in the DeepSeek line compresses the whole thing into a latent vector.\nCache quantization trades a little numerical accuracy for roughly double the capacity, and transformers implements it:\n# requires: pip install optimum-quanto\nout = model.generate(\n    **inputs,\n    do_sample=False,\n    max_new_tokens=20,\n    cache_implementation=\"quantized\",\n    cache_config={\"nbits\": 4, \"backend\": \"quanto\"},\n)\nprint(tokenizer.decode(out[0], skip_special_tokens=True))\nTwo arguments replace the default cache with a quantized one.\nThe KV values are stored at reduced precision, which reduces memory at the cost of quantizing and dequantizing on every access.\nThe backend also requires the group size to divide the model's head dimension evenly, so an unusual architecture can reject the config outright.\nOn short contexts, that overhead can make things slower rather than faster, so it is best used when running low on memory.\nThe cache is freed with the request\nEverything above happens inside one call. The engine frees those blocks when the request finishes, so a 20-turn chat prefills turns 1 through 19 again on turn 20, at full cost.\nYou can see the alternative by keeping the cache alive yourself across turns.\nimport torch\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache\n\nmodel_id = \"HuggingFaceTB/SmolLM2-360M-Instruct\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_id, dtype=torch.bfloat16, device_map=\"auto\"\n)\n\npast_key_values = DynamicCache(config=model.config)\nmessages = []\n\nquestions = [\"What is the capital of France?\", \"And its population?\"]\n\nfor prompt in questions:\n    # Add to the history\n    messages.append({\"role\": \"user\", \"content\": prompt})\n\n   # Tokenize\n    inputs = tokenizer.apply_chat_template(\n        messages,\n        add_generation_prompt=True,\n        return_tensors=\"pt\", return_dict=True\n    ).to(model.device)\n\n    # Generate\n    input_length = inputs[\"input_ids\"].shape[1]\n    outputs = model.generate(\n         **inputs, do_sample=False,\n         max_new_tokens=64,\n         past_key_values=past_key_values\n    )\n\n    # decode\n    completion = tokenizer.decode(outputs[0, input_length:], skip_special_tokens=True)\n\n    # Append to message history\n    messages.append({\"role\": \"assistant\", \"content\": completion})\n    print(f\"turn tokens in: {input_length} | cache now: {past_key_values.get_seq_length()}\")\n\n# Output:\n\"turn tokens in: 42 | cache now: 55\"\n\"turn tokens in: 71 | cache now: 92\"\nThe\npast_key_values\nobject is created once, outside the loop, and passed into every generate call. That way, the cache is not freed at the end of turn one and is still populated when turn two begins.\nOn each turn, we rebuild the full message list and re-render it through\napply_chat_template\n. The prompt sent on turn two contains everything from turn one plus the new question.\nBecause the cache already holds the tokens from turn one, the model only prefills the new suffix. The printed\ninput_length\ngrows every turn while the actual prefill work does not.\nThe completion is sliced off the generated IDs and appended back into messages. That is what makes the next turn's prompt a strict extension of the last one.\nReuse only works because turn two's token sequence starts with turn one's token sequence,  absolutely identical, bit by bit. If you edit anything earlier in the history, the cache becomes invalid.​\nIn this code demo, the cache belongs to one Python variable in one process. In a serving engine, it belongs to a shared pool that thousands of requests look up against. Let's learn about that next.\nTo dive deeper, we built all of this from scratch (with implementation) in\n​\nPart 12 of the RAG systems course\n​\n, including the prefill and decode split and the cache memory formula, with the tensors printed out at each step.\n​\nRead it here →\n2) Prefix caching\nThe shared pool discussed above comes from one change in behavior.\nWhen a request finishes, the engine keeps its KV blocks in memory instead of freeing them, and leaves them indexed so a later request can find them. That is prefix caching.\nThe index has to enforce the same rule covered in the chat loop, where reuse is only valid if the earlier tokens are identical.\nvLLM does that by storing the cache of 16 tokens by default and identifying each block by a hash over the parent block's hash plus the token IDs inside it.\nChaining the parent hash into the child turns a block lookup into a prefix lookup, since a block only matches if everything before it matched too.\nThe scheduler iterates over the incoming blocks in order and stops at the first miss. A hit increments that block’s reference count, which also pins it against eviction while a request is using it.\nEverything from the miss onward gets fresh allocation and a fresh prefill.\nThe lookup code\nvLLM runs this inside its scheduler, wrapped in the memory management that owns the actual tensors.\nThe code below keeps only the two parts that decide reuse, i.e., the function that turns a token sequence into block keys and the function that walks those keys to work out how much of the prefix it can skip prefilling.\nBLOCK_SIZE = 16\n\ndef block_hashes(token_ids, salt=None):\n    \"\"\"Chain-hash a token sequence into per-block keys.\"\"\"\n\n    hashes, parent = [], hash(salt)\n\n    # Only complete blocks are hashed. A partial tail block is skipped.\n    for start in range(0, len(token_ids) - BLOCK_SIZE + 1, BLOCK_SIZE):\n        block = tuple(token_ids[start : start + BLOCK_SIZE])\n        parent = hash((parent, block))\n        hashes.append(parent)\n\n    return hashes\n\ndef schedule(token_ids, cache):\n\n    \"\"\"Return how many tokens are reusable, and allocate the rest.\"\"\"\n\n    matched_blocks = 0\n\n    for h in block_hashes(token_ids):\n        if h not in cache:\n            break                      # first miss ends all reuse\n        cache[h].ref_count += 1        # pin it against eviction\n        matched_blocks += 1\n\n    reused_tokens = matched_blocks * BLOCK_SIZE\n    to_prefill = token_ids[reused_tokens:]\n\n    return reused_tokens, to_prefill\nThe\nblock_hashes\nmethod slices the token sequence into fixed 16-token blocks. Each block's key folds in the previous block's key through hash((parent, block)), so key number five encodes blocks one through five rather than block five alone.\nThe range stops at len(token_ids) - BLOCK_SIZE + 1, which drops any partial block at the tail. Those tokens are never indexed and get recomputed on every request that ends there.\nThe\nschedule\nmethod iterates over the keys in order and stops on the first missing one. There is no attempt to resume matching later in the sequence, because a later block's key already depends on the earlier one that failed.\nref_count += 1 marks the block as in use. Eviction only touches blocks whose count is zero, which is what stops a running request from having its own cache pulled out from under it.\nWhatever gets matched becomes reused_tokens, and everything after it is prefilled fresh.\nThere's one more important thing in the code we just discussed:\nBLOCK_SIZE = 16\n\ndef block_hashes(token_ids, salt=None):\n    \"\"\"Chain-hash a token sequence into per-block keys.\"\"\"\n\n    hashes, parent = [], hash(salt)\n\n    # Only complete blocks are hashed. A partial tail block is skipped.\n    for start in range(0, len(token_ids) - BLOCK_SIZE + 1, BLOCK_SIZE):\n        block = tuple(token_ids[start : start + BLOCK_SIZE])\n        parent = hash((parent, block))\n        hashes.append(parent)\n\n    return hashes\nNotice the\nsalt\nargument in the function above.\nWhen two requests send identical text, they produce identical block keys, so they end up pointing at the same physical KV blocks in GPU memory. There is one copy of those tensors, and both requests read it.\nThat is the behavior you want when both requests come from the same application.\nBut it may need a decision when they come from different customers. So passing a per-tenant value as the salt changes the first parent hash, so identical text now produces different keys for each tenant and their requests never land on the same blocks.\nThis way, every tenant gets its own copy, which costs memory and hit rate but provides separation.\nImplementation in transformers\ntransformers lets you prefill a prompt once and reuse the resulting cache across several different continuations.\nimport copy\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, StaticCache\n\nmodel_id = \"HuggingFaceTB/SmolLM2-360M-Instruct\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_id, dtype=torch.bfloat16, device_map=\"auto\"\n)\n\nSHARED_PREFIX = \"\"\"You are a careful assistant. \n                   Answer in one short sentence.\"\"\"\n\nprompt_cache = StaticCache(config=model.config, max_cache_len=1024)\n\nprefix_inputs = tokenizer(SHARED_PREFIX, return_tensors=\"pt\")\nprefix_inputs = prefix_inputs.to(model.device)\n\n# Prefill the shared prefix exactly once. No token is sampled here.\nwith torch.no_grad():\n    prompt_cache = model(**prefix_inputs, past_key_values=prompt_cache)\n    prompt_cache = prompt_cache.past_key_values\n\nquestions = [\"What is the capital of France?\", \"Name one ocean.\"]\n\nfor question in questions:\n    inputs = tokenizer(SHARED_PREFIX + question, return_tensors=\"pt\")\n    inputs = inputs.to(model.device)\n\n    # each request gets its own copy\n    past_key_values = copy.deepcopy(prompt_cache)   \n\n    outputs = model.generate(\n        **inputs, past_key_values=past_key_values, do_sample=False\n    )\n    print(tokenizer.decode(outputs[0], skip_special_tokens=True))\nStaticCache\nis used instead of DynamicCache because we need a fixed allocation we can copy around.\nThe\nmodel(...)\ncall is a prefill. No token is sampled here. We run the shared prefix through the model purely to populate the cache, then keep the returned past_key_values.\nInside the loop, each question is concatenated onto the same prefix. The full string is tokenized, so the token IDs for the prefix portion are identical every time, which is exactly the condition the engine's hash chain checks for.\ncopy.deepcopy\ngives each request its own copy of the prefilled cache. Generation mutates the cache in place by appending, so without the copy, the first question would corrupt the prefix for the second. A production engine does not copy the tensors. Instead, it shares the physical blocks and tracks reference counts, which is what makes reuse nearly free instead of proportional to prefix length.\nThe impact of eviction on hit rate\nAs discussed above, only complete blocks get indexed, so a trailing partial block is recomputed every time.\nThis means the block size should be tuned appropriately\nLarger blocks imply fewer table lookups and better memory locality\nSmaller blocks imply finer-grained sharing and less waste at the tail.\nEviction reduces hit rates, as expected.\nThe cache and the running batch draw from the same GPU memory pool, so a larger cache leads to fewer concurrent sequences, and under pressure vLLM drops unreferenced blocks by least recent use.\nMixed traffic makes this worse, because long shared prefixes occupy the most blocks and are the ones whose loss actually hurts.\nBefore you turn this on, you should know two things\nIt saves prefill only, so decode time is unchanged and crediting a whole speedup to the cache will overstate it.\nAnd the hashing itself costs something, so on traffic with genuinely unique prompts, benchmarks have measured a throughput regression rather than a gain.\nThere’s a third problem, which is workload dependent, and it impacts\nRAG\nthe most.\nA RAG prompt includes a system instruction, then retrieved chunks, then the query, and the chunks change per request and change order between requests. Two requests that retrieve the same documents in a different order share nothing at all under the chain hash.\nPrefilling each chunk on its own and stitching the caches together does not work.\nThe stitched tensors carry the wrong positional encoding. No chunk ever attended to any other chunk. And every chunk contributes its own attention sink at what the model thinks is position zero. Making it work needs partial recomputation at the boundaries rather than plain concatenation.\nBtw, the solution already exists in open source.\nLMCache (open-source) implements CacheBlend, wherein, instead of gluing the chunk caches end to end, it reuses them at any position and recomputes only a small subset of tokens, chosen by where the precomputed values deviate most from what full attention would have produced.\nThat subset restores the cross-chunk attention and fixes up the positional encoding, so the output holds at full-prefill quality.\nThis leads to an improvement in the time to first token by roughly two to three times compared to recomputing everything, with the recompute cost pipelined against fetching the cached chunks from slower storage.\nIt plugs into vLLM and reads the chunk boundaries out of your prompt, so retrieval traffic gets reused even when the retrieved documents arrive in a different order each time.\nHere's the repo:\nhttps://github.com/LMCache/LMCache\nTo dive deeper, the\n​\nPart 12 of the RAG course\n​\ncovers the solutions to the problem. It also explains the problem with the obvious repair of prefilling each chunk on its own and stitching the caches together, and the problems with it.\n​\nRead it here →\n​\n3) Prompt caching\nOn a hosted model, you don’t get any block table or the eviction policy. Instead, you get a price sheet over the provider’s own prefix reuse, plus two knobs for control.\nThe cached object is still KV tensors, not your prompt text, and it still requires an exact prefix match on the fully rendered context.\nThe rendered context includes provider-side system content you never wrote, which is part of why the minimum lengths and the invalidation rules look arbitrary from the outside.\nHere's a version of prompt caching demonstrated with code:\nimport anthropic\n\nclient = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY from the environment\n\n# Must clear the model's minimum cacheable length or nothing is cached at all.\nLONG_INSTRUCTIONS = \"You are a precise technical editor. \" * 400\n\ndef ask(question: str):\n    return client.messages.create(\n        model=\"claude-sonnet-4-6\",\n        max_tokens=512,\n        system=[\n            {\n                \"type\": \"text\",\n                \"text\": LONG_INSTRUCTIONS,\n                \"cache_control\": {\"type\": \"ephemeral\"},   # everything above is cacheable\n            }\n        ],\n        messages=[{\"role\": \"user\", \"content\": question}],\n    )\n\nfor question in [\"Summarize section 3.\", \"Now rewrite it for a beginner.\"]:\n    resp = ask(question)\n    u = resp.usage\n    print(\n        f\"write={u.cache_creation_input_tokens} \"\n        f\"read={u.cache_read_input_tokens} \"\n        f\"uncached={u.input_tokens}\"\n    )\n\n# Output:\n\"write=2823  read=0     uncached=14\"\n\"write=0     read=2823  uncached=17\"\nOnly one line in that snippet touches the cache.\nWhere you specify\ncache_control\ndecides which part of the request gets an entry written for it, and the usage counters tell you whether a later call read that entry back.\nThe marker is attached to the last block you want covered, not to a range. It writes one cache entry spanning everything from the start of the request up to and including that block.\nThe user message sits below the marker, so it stays outside the cached region since it changes every call, so it must not be inside.\nThe usage counters tell you what's happening under the hood. The first call reports a non-zero cache_creation_input_tokens and a zero read. The second reports the reverse, and the instructions are billed at a tenth of the input rate.\nIf both counters come back as zero, the prefix was below the model's minimum cacheable length, and the request was processed with no caching at all. No error is raised for this.\nIntuitively (and as discussed above), if we move cache_control down onto the user message, the read counter will always be zero, because the marked block changes on every call.\nThe economics of prompt caching\nAnthropic charges 1.25x the base input rate to write an entry and 0.1x to read it, with a higher write multiplier if you want it for a longer time. OpenAI applies the same two multipliers on its current models.\nThe premium cost is recovered in subsequent requests since anything reused inside the TTL will avoid any recomputation.\nA read can only find an entry that some earlier request wrote, and writes happen only at a breakpoint you placed.\nEach call checks your breakpoint, and on a miss it walks backward through a limited number of blocks looking for an older write.\nAnthropic caps that at 20 blocks, so adding more than 20 blocks of conversation between two calls pushes the last write out of range and the hits stop.​\nAll of this is about reusing whatever the provider happened to keep around.\nThe other thing you can do is to prefill your corpus once, deliberately, before any query arrives, and pay to store the resulting cache instead of paying to recompute it.\n​\nPart 13 of the RAG course\n​\nextensively works through that, including the storage arithmetic and how many queries a preloaded cache has to serve before the offline prefill pays for itself.\n​\nRead it here →\n4) Semantic caching\nThe three techniques above save prefill work and still run the model.\nA semantic cache embeds the incoming prompt, runs a nearest-neighbor search over stored prompts, and returns a stored response outright when the similarity exceeds a threshold.\nThat’s why it saves output tokens as well as input. It’s also why every request must bear an embedding round trip, including every miss.\nHere's a working semantic cache demo in a few lines of code:\n# requires: pip install sentence-transformers\nimport numpy as np\nfrom sentence_transformers import SentenceTransformer\n\nencoder = SentenceTransformer(\"all-MiniLM-L6-v2\")\n\nclass SemanticCache:\n    def __init__(self, threshold=0.95):\n        self.threshold = threshold\n        self.vectors = np.empty((0, encoder.get_sentence_embedding_dimension()))\n        self.prompts, self.responses = [], []\n\n    def _embed(self, text):\n        return encoder.encode([text], normalize_embeddings=True)[0]\n\n    def lookup(self, prompt):\n        vec = self._embed(prompt)\n        if len(self.prompts) == 0:\n            return None, 0.0, vec\n        scores = self.vectors @ vec           # cosine sim, vectors are unit length\n        best = int(np.argmax(scores))\n        if scores[best] >= self.threshold:\n            return self.responses[best], float(scores[best]), vec\n        return None, float(scores[best]), vec\n\n    def store(self, prompt, response, vec):\n        self.vectors = np.vstack([self.vectors, vec])\n        self.prompts.append(prompt)\n        self.responses.append(response)\n\ncache = SemanticCache(threshold=0.95)\n\ndef answer(prompt, call_model):\n    hit, score, vec = cache.lookup(prompt)\n    if hit is not None:\n        return hit, f\"HIT  (score {score:.3f})\"\n    response = call_model(prompt)             # the expensive path\n    cache.store(prompt, response, vec)\n    return response, f\"MISS (best {score:.3f})\"\n\n# Stand in for the model so this runs without an API key.\nfake_model = lambda p: f\"<answer for {p!r}>\"\n\nfor q in [\"How do I reset my password?\",\n          \"How can I reset my password?\",\n          \"Is the API rate limited?\"]:\n    _, status = answer(q, fake_model)\n    print(f\"{status}  {q}\")\n\n\n# Output:\n\"MISS (best 0.000)  How do I reset my password?\"\n\"HIT  (score 0.961)  How can I reset my password?\"\n\"MISS (best 0.112)  Is the API rate limited?\"\nEvery method in the class above maps onto a decision you have to make in production:\nnormalize_embeddings=True\nmakes every vector unit length, which lets\nself.vectors @ vec\ncompute cosine similarity as a plain dot product. If you skip the normalization, the scores cannot be compared across prompts of different lengths.\nlookup returns the embedding alongside the result, so answer can store it later without re-embedding. That matters because the embedding is paid on every request, hit or miss, and computing it twice doubles the standing cost of having a cache at all.\nThe brute-force argmax is fine for a demo and wrong at scale. Once you are past a few thousand entries, this becomes an approximate nearest neighbor index, which introduces its own recall setting on top of the threshold.\nstore is called only on the miss path, after the model has answered. Nothing validates that answer before it becomes the response for every future prompt that scores above the threshold. This highlights the biggest risk with this technique. The cache has no notion of whether the stored response was correct, only of whether the new prompt looks similar to the old one.\nThe code below depicts the last point:\npairs = [\n    (\"How do I reset my password?\", \"How can I reset my password?\"),\n    (\"Is the API rate limited?\",     \"Is the API not rate limited?\"),\n    (\"Refund policy for annual plans\", \"Refund policy for monthly plans\"),\n]\n\nfor a, b in pairs:\n    va, vb = encoder.encode([a, b], normalize_embeddings=True)\n    print(f\"{float(va @ vb):.3f}   {a!r}  vs  {b!r}\")\nThis is the output we get:\n0.961   'How do I reset my password?'  vs  'How can I reset my password?'\n0.952   'Is the API rate limited?'  vs  'Is the API not rate limited?'\n0.887   'Refund policy for annual plans'  vs  'Refund policy for monthly plans'\nThe first pair is a genuine paraphrase and should share an answer.\nThe second pair differs by one negation and needs opposite answers.\nThe third differs by one operational value and needs different answers.\nDespite some mismatches, the scores for all three are close together. The paraphrase and the negation are separated by less than a hundredth of a point, which is far too thin a margin to hold across real traffic.\nIf you increase it, the hit rate collapses while you keep paying for embeddings on every call.\nIf you decrease it, the hit rate climbs alongside the rate of confidently wrong answers.\nPublished defaults range from 0.75 to 0.97 depending on who you ask, which tells you it’s a property of your traffic rather than a value to copy.\nThis is not a fully reliable technique per se since some failures (as demonstrated above) can bypass any threshold value, because they come from what embeddings represent.\nRecap of all four techniques\nThree of the four techniques discussed above are correctness-neutral, so their misses show up in cost and latency and nowhere else.\nThe semantic cache works in a different way, so hit rate is not the right metric to report here.\nThere is a fifth, lesser-used layer as well. It's exact-match response cache that returns a stored answer when the request is byte identical. It saves input and output like a semantic cache and carries no false positive risk, because it does no similarity matching at all. You just measure your byte-identical repeat rate before reaching for embeddings. There are problems, of course, as you can probably identify by now. Post them in replies.\nTakeaways for production\nEvery technique has some failure point that you should note before using them in production:\nIf you have any variable in the front of the prompt, like A timestamp, request id or user name in the system prompt, this invalidates every block after it. Always put stable content first, variable content last, and a marker on the boundary.\nTool schemas are usually placed ahead of the system prompt, so a reorder can invalidate the whole cache.\nCheck the settings that get rendered into the prompt. On Anthropic, toggling web search, citations, thinking config, or tool_choice rewrites the prompt text and invalidates downstream blocks. A/B testing two reasoning efforts splits your cache in two.\nSummarizing history rewrites the prefix, so the next call pays full price on cold tokens. Truncating tool outputs in place keeps the prefix byte-identical and the cache alive.\nCache entries are keyed to a model, so routing to a cheaper one still prefills the whole accumulated history at cold rates.\nTo determine exactly where two prompts stop matching, compare their token IDs directly rather than the text you logged. Here's a demonstration:\nmessages_turn_1 = [{\"role\": \"user\", \"content\": \"What is the capital of France?\"}]\nmessages_turn_2 = [{\"role\": \"system\", \"content\": \"Today is Tuesday.\"},\n                   {\"role\": \"user\", \"content\": \"What is the capital of France?\"}]\n\n# tokenize=True is the default and returns a plain list of token ids\na = tokenizer.apply_chat_template(messages_turn_1)\nb = tokenizer.apply_chat_template(messages_turn_2)\n\nshared = 0\nfor x, y in zip(a, b):\n    if x != y:\n        break\n    shared += 1\n\nprint(f\"shared prefix: {shared} tokens of {len(a)} and {len(b)}\")\nprint(f\"first divergence at index {shared}: {a[shared:shared+8]} vs {b[shared:shared+8]}\")\n\n# Output:\n\"\"\"\nshared prefix: 3 tokens of 35 and 26\ndiverges at index 3\n  turn 1: [2683, 418, 253, 11173, 9042, 14260] You are a helpful AI assistant\n  turn 2: [11814, 314, 27758, 30, 2, 198] Today is Tuesday.<|im_end|>\n\"\"\"\nTwo prompts that look identical in your logs can differ by a beginning-of-sequence (BOS) token, a trailing newline, or a re-serialized tool schema.\nComparing token IDs instead of rendered text finds the exact index where reuse stops, and decoding the few IDs on either side usually finds the exact text.\nThe run above shows a common one.\nTurn one specified no system message, so the chat template filled in the model's default, and the two prompts looked different at index 3, so no reuse was possible.\nThe first three layers cover one idea, applied at three scopes.\nThe KV cache holds attention state for the duration of a single request.\nPrefix caching keeps that state after the request ends so a later request can look it up.\nPrompt caching is a provider running prefix caching on their own hardware and charging a separate rate for the part you reuse.\nThe semantic cache works differently. It stores response text keyed by embedding similarity, so if there's a hit, it skips the model entirely and saves output tokens along with input tokens. A hit can also be wrong, and it returns with a normal success status when it is.\nAs further reading:\n​\nRAG course Part 12 covers the prefill and decode split and why prefix caching underperforms on RAG, with implementations →\n​\n​\nRAG course Part 13 covers preloading a corpus into a cache before any query arrives →\n​\n​\nRAG course Parts 14 and 15 cover cache compression that has to run before a query exists, plus the training-based approaches and what production involves →\n👉 Over to you: which of these four layers has cost you the most debugging time?\nGood day!\nPublished on Aug 28, 2026\nComments\nShare\nCopy link\nShare to X\nShare to Facebook\nShare to Linkedin\nCopied\nPrevious\nGrok Bot Masterclass\nNext\nPreloading Knowledge Into a Model Instead of Retrieving It (Part C)",
  "metadata": {
    "title": "KV vs Prefix vs Prompt vs Semantic Caching",
    "description": "...explained with best practices in production.",
    "keywords": null,
    "author": null,
    "og:site_name": "Daily Dose of Data Science",
    "og:type": "article",
    "og:title": "KV vs Prefix vs Prompt vs Semantic Caching",
    "og:description": "...explained with best practices in production.",
    "og:url": "https://www.dailydoseofds.com/p/kv-vs-prefix-vs-prompt-vs-semantic-caching/",
    "og:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/clipboard-image-1788111664.png",
    "og:image:width": "1200",
    "og:image:height": "480",
    "twitter:card": "summary_large_image",
    "twitter:title": "KV vs Prefix vs Prompt vs Semantic Caching",
    "twitter:description": "...explained with best practices in production.",
    "twitter:url": "https://www.dailydoseofds.com/p/kv-vs-prefix-vs-prompt-vs-semantic-caching/",
    "twitter:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/clipboard-image-1788111664.png",
    "twitter:label1": "Written by",
    "twitter:data1": "Avi Chawla",
    "twitter:label2": "Filed under",
    "twitter:data2": "LLMOps",
    "twitter:site": "@_avichawla",
    "article:published_time": "2026-08-28T18:08:00.000Z",
    "article:modified_time": "2026-08-30T18:11:52.000Z",
    "article:tag": "LLMOps"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/9c3f9ccc7fe7/code/body-000.txt",
      "chars": 239,
      "language": "text"
    },
    {
      "file": "attachments/9c3f9ccc7fe7/code/body-001.txt",
      "chars": 976,
      "language": "text"
    },
    {
      "file": "attachments/9c3f9ccc7fe7/code/body-002.txt",
      "chars": 268,
      "language": "text"
    },
    {
      "file": "attachments/9c3f9ccc7fe7/code/body-003.txt",
      "chars": 1302,
      "language": "text"
    },
    {
      "file": "attachments/9c3f9ccc7fe7/code/body-004.txt",
      "chars": 930,
      "language": "text"
    },
    {
      "file": "attachments/9c3f9ccc7fe7/code/body-005.txt",
      "chars": 446,
      "language": "text"
    },
    {
      "file": "attachments/9c3f9ccc7fe7/code/body-006.txt",
      "chars": 1257,
      "language": "text"
    },
    {
      "file": "attachments/9c3f9ccc7fe7/code/body-007.txt",
      "chars": 1001,
      "language": "text"
    },
    {
      "file": "attachments/9c3f9ccc7fe7/code/body-008.txt",
      "chars": 1868,
      "language": "text"
    },
    {
      "file": "attachments/9c3f9ccc7fe7/code/body-009.txt",
      "chars": 362,
      "language": "text"
    },
    {
      "file": "attachments/9c3f9ccc7fe7/code/body-010.txt",
      "chars": 224,
      "language": "text"
    },
    {
      "file": "attachments/9c3f9ccc7fe7/code/body-011.txt",
      "chars": 868,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "bytes": 11344,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/000.jpg",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "alt": "Avi Chawla",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-36.png",
      "bytes": 335628,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/001.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-36.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-37.png",
      "bytes": 305200,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/003.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-37.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-38.png",
      "bytes": 300063,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/005.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-38.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-39.png",
      "bytes": 203162,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/007.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-39.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-40.png",
      "bytes": 296746,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/009.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-40.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-55.png",
      "bytes": 247406,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/011.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-55.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-41.png",
      "bytes": 252842,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/013.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-41.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-42.png",
      "bytes": 185403,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/015.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-42.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-43.png",
      "bytes": 282271,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/017.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-43.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/rag-diagram-new.gif",
      "bytes": 315345,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/019.gif",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/rag-diagram-new.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-44.png",
      "bytes": 134333,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/021.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-44.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-45.png",
      "bytes": 312883,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/023.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-45.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-46.png",
      "bytes": 233928,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/025.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-46.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-47.png",
      "bytes": 182474,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/027.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-47.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-48.png",
      "bytes": 260932,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/029.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-48.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-49.png",
      "bytes": 305734,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/031.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-49.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-50.png",
      "bytes": 308578,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/033.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-50.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-51.png",
      "bytes": 282781,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/035.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-51.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-52.png",
      "bytes": 282620,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/037.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-52.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-53.png",
      "bytes": 210854,
      "reused": true,
      "local_path": "attachments/9c3f9ccc7fe7/images/039.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-53.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-54.png",
      "bytes": 340041,
      "reused": false,
      "local_path": "attachments/9c3f9ccc7fe7/images/body-88e854914e6d31d53945.png",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/08/image-54.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/media/2026/08/kv-caching-final.mp4",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/media/2026/08/kv-caching-final.mp4",
      "status": "invalid_or_wrong_resolution",
      "kind": "direct_video",
      "bytes": 6041941,
      "reused": true,
      "content_type": "video/mp4",
      "valid": true,
      "height": 732,
      "width": 1256,
      "duration": "46.516667",
      "local_path": "attachments/9c3f9ccc7fe7/videos/repair-7edce4725d464569b19c.mp4",
      "title": "",
      "retried_at": "2026-10-20T18:11:35Z"
    }
  ],
  "links": [
    {
      "href": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/media/2026/08/kv-caching-final.mp4",
      "label": "",
      "type": "video"
    },
    {
      "href": "https://www.dailydoseofds.com/building-rag-systems-course-part-12-with-implementation/",
      "label": "​",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/",
      "label": "RAG",
      "type": "a"
    },
    {
      "href": "https://github.com/LMCache/LMCache?ref=dailydoseofds.com",
      "label": "https://github.com/LMCache/LMCache",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/building-rag-systems-course-part-13-with-implementation/",
      "label": "​",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/building-rag-systems-course-part-14-with-implementation/",
      "label": "​",
      "type": "a"
    },
    {
      "href": "https://x.com/intent/tweet?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fkv-vs-prefix-vs-prompt-vs-semantic-caching%2F&text=KV%20vs%20Prefix%20vs%20Prompt%20vs%20Semantic%20Caching",
      "label": "Share to X",
      "type": "a"
    },
    {
      "href": "https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fkv-vs-prefix-vs-prompt-vs-semantic-caching%2F",
      "label": "Share to Facebook",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fkv-vs-prefix-vs-prompt-vs-semantic-caching%2F",
      "label": "Share to Linkedin",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/p/grok-bot-masterclass/",
      "label": "Previous Grok Bot Masterclass",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/building-rag-systems-course-part-15-with-implementation/",
      "label": "Next Preloading Knowledge Into a Model Instead of Retrieving It (Part C)",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:54Z",
  "source_url": "https://www.dailydoseofds.com/p/kv-vs-prefix-vs-prompt-vs-semantic-caching/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 32209,
  "expected_body_images": 22,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "invalid_or_wrong_resolution",
  "raw_html": "attachments/9c3f9ccc7fe7/pages/source.html",
  "local_html": "attachments/9c3f9ccc7fe7/pages/article.html",
  "body_base_path": "attachments/9c3f9ccc7fe7/pages/",
  "body_markdown_path": "attachments/9c3f9ccc7fe7/pages/article.md"
}