{ "source_file": "0930", "source_platform": "Substack", "author": "Sebastian Raschka / Ahead of AI", "title_listed": "Understanding and Coding the KV Cache in LLMs from Scratch", "keywords": "KV缓存,推理优化", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://magazine.sebastianraschka.com/p/coding-the-kv-cache-in-llms", "publish_date": "2026-06", "corpus_stage": "推理部署", "notes": "KV缓存从零实现", "id": "c9d8040bdb31", "collected_at": "2026-10-01T20:32:45", "title": "Understanding and Coding the KV Cache in LLMs from Scratch", "body_markdown": "KV caches are one of the most critical techniques for efficient inference in LLMs in production. KV caches are an important component for compute-efficient LLM inference in production. This article explains how they work conceptually and in code with a from-scratch, human-readable implementation.\nIt's been a while since I shared a technical tutorial explaining fundamental LLM concepts. As I am currently recovering from an injury and working on a bigger LLM research-focused article, I thought I'd share a tutorial article on a topic several readers asked me about (as it was\nnot\nincluded in my\nBuilding a Large Language Model From Scratch\nbook).\nHappy reading!\nOverview\nIn short, a KV cache stores intermediate key (K) and value (V) computations for reuse during inference (after training), which results in a substantial speed-up when generating text. The downside of a KV cache is that it adds more complexity to the code, increases memory requirements (the main reason I initially didn't include it in the book), and can't be used during training. However, the inference speed-ups are often well worth the trade-offs in code complexity and memory when using LLMs in production.\nWhat Is a KV Cache?\nImagine the LLM is generating some text. Concretely, suppose the LLM is given the following prompt: \"Time\". As you may already know, LLMs generate one word (or token) at a time, and the two following text generation steps may look as illustrated in the figure below:\nFigure 1: The diagram illustrates how an LLM generates text one token at a time. Starting with the prompt “Time”, the model generates the next token “flies.” In the next step, the full sequence “Time flies” is reprocessed to generate the token “fast”.\nNote that there is some redundancy in the generated LLM text outputs, as highlighted in the next figure:\nFigure 2: This figure highlights the repeated context (”Time flies”) that must be reprocessed by the LLM at each generation step. Since the LLM does not cache intermediate key/value states, it re-encodes the full sequence every time a new token (e.g., “fast”) is generated.\nWhen we implement an LLM text generation function, we typically only use the last generated token from each step. However, the visualization above highlights one of the main inefficiencies on a conceptual level. This inefficiency (or redundancy) becomes more clear if we zoom in on the attention mechanism itself. (If you are curious about attention mechanisms, you can read more in Chapter 3 of my\nBuild a Large Language Model (From Scratch)\nbook or my\nUnderstanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs\narticle).\nUnderstanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs\nJanuary 14, 2024\nRead full story\nThe following figure shows an excerpt of an attention mechanism computation that is at the core of an LLM. Here, the input tokens (\"Time\" and \"flies\") are encoded as 3-dimensional vectors (in reality, these vectors are much larger, but this would make it challenging to fit them into a small figure). The matrices\nW\nare the weight matrices of the attention mechanism that transform these inputs into key, value, and query vectors.\nThe figure below shows an excerpt of the underlying attention score computation with the key and value vectors highlighted:\nFigure 3: This figure illustrates how the LLM derives key (k) and value (v) vectors from token embeddings during attention computation. Each input token (e.g., “Time” and “flies”) is projected using learned matrices W_k and W_v to obtain its corresponding key and value vectors.\nAs mentioned earlier, LLMs generate one word (or token) at a time. Suppose the LLM generated the word \"fast\" so that the prompt for the next round becomes \"Time flies fast\". This is illustrated in the next figure below:\nFigure 4: This diagram shows how the LLM recomputes key and value vectors for previously seen tokens (”Time” and “flies”) during each generation step. When generating the third token (”fast”), the model recomputes the same k(1)/v(1) and k(2)/v(2) vectors again, rather than reusing them. This repeated computation highlights the inefficiency of not using a KV cache during autoregressive decoding.\nAs we can see, based on comparing the previous 2 figures, the keys and value vectors for the first two tokens are exactly the same, and it would be wasteful to recompute them in each next-token text generation round.\nNow, the idea of the KV cache is to implement a caching mechanism that stores the previously generated key and value vectors for reuse, which helps us to avoid these unnecessary recomputations.\nHow LLMs Generate Text (Without and With a KV Cache)\nAfter we went over the basic concept in the previous section, let's go into a bit more detail before we look at a concrete code implementation. If we have a text generation process\nwithout\nKV cache for \"Time flies fast\", we can think of it as follows:\nNotice the redundancy: tokens \"Time\" and \"flies\" are recomputed at every new generation step. The KV cache resolves this inefficiency by storing and reusing previously computed key and value vectors:\nInitially, the model computes and caches key and value vectors for the input tokens.\nFor each new token generated, the model only computes key and value vectors for that specific token.\nPreviously computed vectors are retrieved from the cache to avoid redundant computations.\nThe table below summarizes the computation and caching steps and states:\nThe benefits here are that\n\"Time\"\nis computed once and reused twice, and\n\"flies\"\nis computed once and reused once. (It's a short text example for simplicity, but it should be intuitive to see that the longer the text, the more we get to reuse already computed keys and values, which increases the generation speed.)n speed.)\nThe following figure illustrates generation step 3 with and without a KV cache side by side.\nFigure 5: Comparing text generation with and without a KV cache part 1/2. Without cache. Key and value vectors are recomputed for each token step, which results in redundant operations.\nFigure 6. Comparing text generation with and without a KV cache part 2/2. With KV cache. Previously computed keys and values are retrieved from the KV cache to avoid recomputation for faster generation.\nSo, if we want to implement a KV cache in code, all we have to do is compute the keys and values as usual but then store them so that we can retrieve them in the next round. The next section illustrates this with a concrete code example.\nImplementing a KV Cache from Scratch\nThere are many ways to implement a KV cache, with the main idea being that we only compute the key and value tensors for the newly generated tokens in each generation step.\nI opted for a simple one that emphasizes code readability. I think it's easiest to just scroll through the code changes to see how it's implemented.\nThere are two files I shared on GitHub, which are self-contained Python scripts that implement an LLM with and without KV cache from scratch:\ngpt_ch04.py\n: Self-contained code taken from Chapters 3 and 4 of my\nBuild a Large Language Model (From Scratch)\nbook to implement the LLM and run the simple text generation function\ngpt_with_kv_cache.py\n: The same as above, but with the necessary changes made to implement the KV cache.\nTo read through the KV cache-relevant code modifications, you can either:\na. Open the\ngpt_with_kv_cache.py\nfile and look out for the\n# NEW\nsections that mark the new changes:\nb. Check out the two code files via a file diff tool of your choice to compare the changes:\nIn addition, to summarize the implementation details, there's a short walkthrough in the following subsections.\n1. Registering the Cache Buffers\nInside the\nMultiHeadAttention\nconstructor, we add two buffers,\ncache_k\nand\ncache_v\n, which will hold concatenated keys and values across steps:\nself.register_buffer(\"cache_k\", None)\nself.register_buffer(\"cache_v\", None)\n(I made a YouTube video if you want to learn more about buffers:\nUnderstanding PyTorch Buffers\n.)\n2. Forward pass with\nuse_cache\nflag\nNext, we extend the\nforward\nmethod of the\nMultiHeadAttention\nclass to accept a\nuse_cache\nargument:\ndef forward(self, x, use_cache=False):\n b, num_tokens, d_in = x.shape\n​\n keys_new = self.W_key(x) # Shape: (b, num_tokens, d_out)\n values_new = self.W_value(x)\n queries = self.W_query(x)\n #...\n​\n if use_cache:\n if self.cache_k is None:\n self.cache_k, self.cache_v = keys_new, values_new\n else:\n self.cache_k = torch.cat([self.cache_k, keys_new], dim=1)\n self.cache_v = torch.cat([self.cache_v, values_new], dim=1)\n keys, values = self.cache_k, self.cache_v\n else:\n keys, values = keys_new, values_new\nThe storage and retrieval of keys and values here implements the core idea of the KV cache.\nStoring\nConcretely, after the cache is initialized via the if\nself.cache_k is None: ...\n, we add the newly generated keys and values via\nself.cache_k = torch.cat(...)\nand\nself.cache_v = torch.cat(...)\nto the cache, respectively.\nRetrieving\nThen,\nkeys, values = self.cache_k, self.cache_v\nretrieves the stored values and keys from the cache.\nAnd that's basically it: the core store & retrieve mechanism of a KV cache. The following sections, 3 and 4, just take care of minor implementation details.\n3. Clearing the Cache\nWhen generating text, we have to remember to reset both the keys and value buffers between two separate text-generation calls. Otherwise, the queries of a new prompt will attend to stale keys left over from the previous sequence, which causes the model to rely on irrelevant context and produce incoherent output. To prevent this, we add a\nreset_kv_cache\nmethod to the\nMultiHeadAttention\nclass that we can use between text-generation calls later:\ndef reset_cache(self):\n self.cache_k, self.cache_v = None, None\n4. Propagating\nuse_cache\nin the Full Model\nWith the changes to the\nMultiHeadAttention\nclass in place, we now modify the\nGPTModel\nclass. First, we add a position tracking for the token indices to the instructor:\nself.current_pos = 0\nThis is a simple counter that remembers how many tokens the model has already cached during an incremental generation session.\nThen, we replace the one-liner block call with an explicit loop, passing\nuse_cache\nthrough each transformer block:\ndef forward(self, in_idx, use_cache=False):\n # ...\n \n if use_cache:\n pos_ids = torch.arange(\n self.current_pos, self.current_pos + seq_len, \n device=in_idx.device, dtype=torch.long\n )\n self.current_pos += seq_len\n else:\n pos_ids = torch.arange(\n 0, seq_len, device=in_idx.device, dtype=torch.long\n )\n \n pos_embeds = self.pos_emb(pos_ids).unsqueeze(0)\n x = tok_embeds + pos_embeds\n # ...\n for blk in self.trf_blocks:\n x = blk(x, use_cache=use_cache)\nWhat happens above if we set\nuse_cache=True\nis that we start at the\nself.current_pos\nand count\nseq_len\nsteps. Then, bump the counter so the next decoding call continues where we left off.\nThe reason for the\nself.current_pos\ntracking is that new queries must line up directly after the keys and values that are already stored. Without using a counter, every new step would start at position 0 again, so the model would treat the new tokens as if they overlapped the earlier ones. (Alternatively, we could also keep track via an\noffset = block.att.cache_k.shape[1]\n.)\nThe above change then also requires a small modification to the\nTransformerBlock\nclass to accept the\nuse_cache\nargument:\ndef forward(self, x, use_cache=False):\n # ...\n self.att(x, use_cache=use_cache)\nLastly, we add a model-level reset to\nGPTModel\nto clear all block caches at once for our convenience:\ndef reset_kv_cache(self):\n for blk in self.trf_blocks:\n blk.att.reset_cache()\n self.current_pos = 0\n5. Using the Cache in Generation\nWith the changes to the\nGPTModel\n,\nTransformerBlock\n, and\nMultiHeadAttention\n, finally, here's how we use the KV cache in a simple text generation function:\ndef generate_text_simple_cached(\n model, idx, max_new_tokens, use_cache=True\n ):\n model.eval()\n​\n ctx_len = model.pos_emb.num_embeddings # max sup. len., e.g. 1024\n if use_cache:\n # Init cache with full prompt\n model.reset_kv_cache()\n with torch.no_grad():\n logits = model(idx[:, -ctx_len:], use_cache=True)\n​\n for _ in range(max_new_tokens):\n # a) pick the token with the highest log-probability \n next_idx = logits[:, -1].argmax(dim=-1, keepdim=True)\n # b) append it to the running sequence\n idx = torch.cat([idx, next_idx], dim=1)\n # c) feed model only the new token\n with torch.no_grad():\n logits = model(next_idx, use_cache=True)\n else:\n for _ in range(max_new_tokens):\n with torch.no_grad():\n logits = model(idx[:, -ctx_len:], use_cache=False)\n next_idx = logits[:, -1].argmax(dim=-1, keepdim=True)\n idx = torch.cat([idx, next_idx], dim=1)\n​\n return idx\nNote that we only feed the model the new token in c) via\nlogits = model(next_idx, use_cache=True)\n. Without caching, we feed the model the whole input\nlogits = model(idx[:, -ctx_len:], use_cache=False)\nas it has no stored keys and values to reuse.\nA Simple Performance Comparison\nAfter covering the KV cache on a conceptual level, the big question is how well it actually performs in practice on a small example. To give the implementation a try, we can run the two aforementioned code files as Python scripts, which will run the small 124 M parameter LLM to generate 200 new tokens (given a 4-token prompt \"Hello, I am\" to start with):\npip install -r https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/refs/heads/main/requirements.txt\n​\npython gpt_ch04.py\n​\npython gpt_with_kv_cache.py\nOn a Mac Mini with M4 chip (CPU), the results are as follows:\nSo, as we can see, we already get a ~5x speed-up with a small 124 M parameter model and a short 200-token sequence length. (Note that this implementation is optimized for code readability and not optimized for CUDA or MPS runtime speed, which would require pre-allocating tensors instead of reinstating and concatenating them.)\nNote:\nThe model generates \"gibberish\" in both cases, i.e., text that looks like this:\nOutput text: Hello, I am Featureiman Byeswickattribute argue logger Normandy Compton analogous bore ITVEGIN ministriesysics Kle functional recountrictionchangingVirgin embarrassedgl ...\nThis is because we haven't trained the model yet. The next chapter trains the model, and you can use the KV cache on the trained model (however, the KV cache is only meant to be used during inference) to generate coherent text. Here, we are using the untrained model to keep the code simple(r).\nWhat's more important, though, is that both the\ngpt_ch04.py\nand\ngpt_with_kv_cache.py\nimplementations produce exactly the same text. This tells us that the KV cache is implemented correctly -- it is easy to make indexing mistakes that can lead to divergent results.\nThanks for reading Ahead of AI! Subscribe for free to receive new posts and support my work.\nSubscribe\nKV cache Advantages and Disadvantages\nAs sequence length increases, the benefits and downsides of a KV cache become more pronounced in the following ways:\n[Good]\nComputational efficiency increases\n: Without caching, the attention at step\nt\nmust compare the new query with\nt\nprevious keys, so the cumulative work scales quadratically, O(n²). With a cache, each key and value is computed once and then reused, reducing the total per-step complexity to linear, O(n).\n[Bad]\nMemory usage increases linearly\n: Each new token appends to the KV cache. For long sequences and larger LLMs, the cumulative KV cache grows larger, which can consume a significant or even prohibitive amount of (GPU) memory. As a workaround, we can truncate the KV cache, but this adds even more complexity (but again, it may well be worth it when deploying LLMs.)\nOptimizing the KV Cache Implementation\nWhile my conceptual implementation of a KV cache above helps with clarity and is mainly geared towards code readability and educational purposes, deploying it in real-world scenarios (especially with larger models and longer sequence lengths) requires more careful optimization.\nCommon Pitfalls When Scaling the Cache\nMemory fragmentation and repeated allocations\n: Continuously concatenating tensors via\ntorch.cat\n, as shown earlier, leads to performance bottlenecks due to frequent memory allocation and reallocation.\nLinear growth in memory usage\n: Without proper handling, the KV cache size becomes impractical for very long sequences.\nTip 1: Pre-allocate Memory\nRather than concatenating tensors repeatedly, we could pre-allocate a sufficiently large tensor based on the expected maximum sequence length. This ensures consistent memory use and reduces overhead. In pseudo-code, this may look like as follows:\n# Example pre-allocation for keys and values\nmax_seq_len = 1024 # maximum expected sequence length\ncache_k = torch.zeros(\n (batch_size, num_heads, max_seq_len, head_dim), device=device\n)\ncache_v = torch.zeros(\n (batch_size, num_heads, max_seq_len, head_dim), device=device\n)\nDuring inference, we can then simply write into slices of these pre-allocated tensors.\nTip 2: Truncate Cache via Sliding Window\nTo avoid blowing up our GPU memory, we can implement a sliding window approach with dynamic truncation. Via the sliding window, we maintain only the last\nwindow_size\ntokens in the cache:\n# Sliding window cache implementation\nwindow_size = 512\ncache_k = cache_k[:, :, -window_size:, :]\ncache_v = cache_v[:, :, -window_size:, :]\nOptimizations in Practice\nYou can find these optimizations in the\ngpt_with_kv_cache_optimized.py\nfile.\nOn a Mac Mini with an M4 chip (CPU), with a 200-token generation and a window size equal to the LLM's context length (to guarantee the same results and thus a fair comparison) below, the code runtimes compare as follows:\nUnfortunately, the speed advantages disappear on CUDA devices as this is a tiny model, and the device transfer and communication outweigh the benefits of a KV cache for this small model.\nConclusion\nAlthough caching introduces additional complexity and memory considerations, the noticeable gains in efficiency typically outweigh these trade-offs, especially in production environments.\nRemember, while I prioritized code clarity and readability over efficiency here, the takeaway is that practical implementations often require thoughtful optimizations, such as pre-allocating memory or applying a sliding-window cache to manage memory growth effectively. In that sense, I hope this article turned out to be informative.\nFeel free to experiment with these techniques, and happy coding!\nBonus: KV Caches in Qwen3 and Llama 3\nAfter adding KV caches to my from-scratch implementations of Qwen3 (0.6 B) and Llama 3 (1 B), I ran additional experiments comparing the model runtimes with and without KV cache. Note that I opted for the torch.cat approach mentioned above rather than pre-allocating the KV cache tensors as described in the\nOptimizing the KV Cache Implementation\nsection. Since Llama 3 and Qwen3 have very large supported context sizes (131k and 41k tokens, respectively), the pre-allocated tensors consume ~8 GB of additional memory, which is quite expensive.\nMoreover, because I am using the more memory-efficient\ntorch.cat\napproach to creating the tensors on the fly, I moved the KV cache outside the model to compile the model with\ntorch.compile\nfor a computational efficiency boost.\nThe codes can be found here:\nqwen3.py\n|\nREADME\nllama3.py\n|\nREADME\nAnd the performances are shown below.\nAs we can see, on CPUs, the KV cache results in the most substantial speed-up. And compilation boosts that performance even further. However, on a GPU, the best performance can be achieved with the regular compiled model, which is likely because we don’t pre-allocate the tensors on the GPU, and the models are relatively small.\nThis magazine is a personal passion project. To support me as an independent researcher, please consider purchasing a copy of my book,\nBuild a Large Language Model (From Scratch) book\n, or signing up for a\npaid subscription\n.\nBuild a Large Language Model (From Scratch) now\navailable on Amazon\nIf you read the book and have a few minutes to spare, I'd really appreciate a\nbrief review\n. It helps us authors a lot!\nYour support means a great deal! Thank you!", "metadata": { "title": "Understanding and Coding the KV Cache in LLMs from Scratch", "description": "KV caches are one of the most critical techniques for efficient inference in LLMs in production.", "keywords": null, "author": "Sebastian Raschka, PhD", "og:type": "article", "og:title": "Understanding and Coding the KV Cache in LLMs from Scratch", "og:description": "KV caches are one of the most critical techniques for efficient inference in LLMs in production.", "og:image": "https://substackcdn.com/image/fetch/$s_!lRr3!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1045ae83-d2ba-4b44-ac93-4159d8a610a1_3659x2622.png", "og:url": "https://magazine.sebastianraschka.com/p/coding-the-kv-cache-in-llms", "twitter:title": "Understanding and Coding the KV Cache in LLMs from Scratch", "twitter:description": "KV caches are one of the most critical techniques for efficient inference in LLMs in production.", "twitter:image": "https://substackcdn.com/image/fetch/$s_!kY7K!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fsebastianraschka.substack.com%2Fapi%2Fv1%2Fpost_preview%2F166106178%2Ftwitter.jpg%3Fversion%3D4", "twitter:card": "summary_large_image", "article:modified_time": "2026-09-13T18:23:42.139Z" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/c9d8040bdb31/code/body-000.txt", "chars": 75, "language": "text" }, { "file": "attachments/c9d8040bdb31/code/body-001.txt", "chars": 583, "language": "text" }, { "file": "attachments/c9d8040bdb31/code/body-002.txt", "chars": 66, "language": "text" }, { "file": "attachments/c9d8040bdb31/code/body-003.txt", "chars": 20, "language": "text" }, { "file": "attachments/c9d8040bdb31/code/body-004.txt", "chars": 558, "language": "text" }, { "file": "attachments/c9d8040bdb31/code/body-005.txt", "chars": 85, "language": "text" }, { "file": "attachments/c9d8040bdb31/code/body-006.txt", "chars": 112, "language": "text" }, { "file": "attachments/c9d8040bdb31/code/body-007.txt", "chars": 1060, "language": "text" }, { "file": "attachments/c9d8040bdb31/code/body-008.txt", "chars": 156, "language": "text" }, { "file": "attachments/c9d8040bdb31/code/body-009.txt", "chars": 281, "language": "text" }, { "file": "attachments/c9d8040bdb31/code/body-010.txt", "chars": 139, "language": "text" } ], "images": [ { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!xNU1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47afb112-1e46-42bf-80b3-b7b11d0a55eb_1804x1820.png", "bytes": 169244, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-45280a1a6ddb0631cf5b.png", "src": "https://substackcdn.com/image/fetch/$s_!xNU1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47afb112-1e46-42bf-80b3-b7b11d0a55eb_1804x1820.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!As0Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda5df468-5b21-4b1f-9ccb-b144dfb2a293_617x618.png", "bytes": 16433, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-d22c6e004f94a37611e5.png", "src": "https://substackcdn.com/image/fetch/$s_!As0Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda5df468-5b21-4b1f-9ccb-b144dfb2a293_617x618.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!3NS4!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69bfee26-ea3b-42a6-8a1a-6b8187852082_738x564.png", "bytes": 2972, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-c6f1361fdb6171542e26.png", "src": "https://substackcdn.com/image/fetch/$s_!3NS4!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69bfee26-ea3b-42a6-8a1a-6b8187852082_738x564.png", "alt": "Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!8eFC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7f30c1-ebb2-4bb0-8cf9-fd89f33825b2_1984x1226.png", "bytes": 78631, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-2a2bc15dee4475721e04.png", "src": "https://substackcdn.com/image/fetch/$s_!8eFC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7f30c1-ebb2-4bb0-8cf9-fd89f33825b2_1984x1226.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!Ewqz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cd206dd-5685-459a-bdc6-45858e0f18f7_2706x2580.png", "bytes": 122678, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-387264aee73ac0a0f051.png", "src": "https://substackcdn.com/image/fetch/$s_!Ewqz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cd206dd-5685-459a-bdc6-45858e0f18f7_2706x2580.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!z-sX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fdd4b41-96e6-40c2-baf3-aedfeee8d1de_741x194.png", "bytes": 6584, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-7c46c766a59bf29ff187.png", "src": "https://substackcdn.com/image/fetch/$s_!z-sX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fdd4b41-96e6-40c2-baf3-aedfeee8d1de_741x194.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!qQBU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f20643e-0942-4b5f-98ec-051d030d127a_736x202.png", "bytes": 7731, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-adc122e2ff6cc6f407ba.png", "src": "https://substackcdn.com/image/fetch/$s_!qQBU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f20643e-0942-4b5f-98ec-051d030d127a_736x202.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!f5RK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea5b59d9-6c1b-4aee-8f65-0a85523e9807_3659x1677.png", "bytes": 56237, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-234b9bcb46839355e078.png", "src": "https://substackcdn.com/image/fetch/$s_!f5RK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea5b59d9-6c1b-4aee-8f65-0a85523e9807_3659x1677.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!a_ei!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F711178d4-cb9c-45dc-bd0c-b8d9f476329c_3659x2622.png", "bytes": 127676, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-25744d41d21e7e63425f.png", "src": "https://substackcdn.com/image/fetch/$s_!a_ei!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F711178d4-cb9c-45dc-bd0c-b8d9f476329c_3659x2622.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!KVf4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3338a3b0-c3ad-4d37-9d3f-15db18db51ff_982x881.png", "bytes": 106103, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-e7b74666611b5448c568.png", "src": "https://substackcdn.com/image/fetch/$s_!KVf4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3338a3b0-c3ad-4d37-9d3f-15db18db51ff_982x881.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!qPKJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccfd3b14-df6e-4988-b986-851b4b207eae_1468x861.png", "bytes": 182436, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-0aa4dce8d53e56b2d802.png", "src": "https://substackcdn.com/image/fetch/$s_!qPKJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccfd3b14-df6e-4988-b986-851b4b207eae_1468x861.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!LWIV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8d75a600-22e3-4edf-b642-44700dc7a2db_743x160.png", "bytes": 4565, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-4a2ad58fe88bc270a620.png", "src": "https://substackcdn.com/image/fetch/$s_!LWIV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8d75a600-22e3-4edf-b642-44700dc7a2db_743x160.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!9fY3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14878ff2-28f2-496f-a5ad-03aa2865640c_742x197.png", "bytes": 5556, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-37a60e3e28413242b53a.png", "src": "https://substackcdn.com/image/fetch/$s_!9fY3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14878ff2-28f2-496f-a5ad-03aa2865640c_742x197.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!-pE8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d3a0bf-4a1a-439a-a83c-9046e1a57515_931x616.png", "bytes": 47093, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-540c520f0b5e0c7f1336.png", "src": "https://substackcdn.com/image/fetch/$s_!-pE8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d3a0bf-4a1a-439a-a83c-9046e1a57515_931x616.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!O7SH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49ce3862-820f-4337-91b7-6b4603ec8f86_876x598.png", "bytes": 40830, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-6ddff7fbc2bf9c185856.png", "src": "https://substackcdn.com/image/fetch/$s_!O7SH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49ce3862-820f-4337-91b7-6b4603ec8f86_876x598.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!woQp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png", "bytes": 121470, "reused": false, "local_path": "attachments/c9d8040bdb31/images/body-1c343ecc9eadd930f4bd.png", "src": "https://substackcdn.com/image/fetch/$s_!woQp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/watch?v=PetlIokI9Ao", "status": "ok", "local_path": "attachments/c9d8040bdb31/videos/PetlIokI9Ao.mp4", "bytes": 31648566, "valid": true, "height": 1080, "width": 1920, "duration": "812.061000" } ], "links": [ { "href": "https://substackcdn.com/image/fetch/$s_!xNU1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47afb112-1e46-42bf-80b3-b7b11d0a55eb_1804x1820.png", "label": "", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!As0Z!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda5df468-5b21-4b1f-9ccb-b144dfb2a293_617x618.png", "label": "", "type": "a" }, { "href": "https://amzn.to/4fqvn0D", "label": "Build a Large Language Model (From Scratch)", "type": "a" }, { "href": "https://magazine.sebastianraschka.com/p/understanding-and-coding-self-attention", "label": "Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!8eFC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7f30c1-ebb2-4bb0-8cf9-fd89f33825b2_1984x1226.png", "label": "", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!Ewqz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cd206dd-5685-459a-bdc6-45858e0f18f7_2706x2580.png", "label": "", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!f5RK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea5b59d9-6c1b-4aee-8f65-0a85523e9807_3659x1677.png", "label": "", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!a_ei!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F711178d4-cb9c-45dc-bd0c-b8d9f476329c_3659x2622.png", "label": "", "type": "a" }, { "href": "https://github.com/rasbt/LLMs-from-scratch/blob/main/ch04/03_kv-cache/gpt_ch04.py", "label": "gpt_ch04.py", "type": "a" }, { "href": "https://github.com/rasbt/LLMs-from-scratch/blob/main/ch04/03_kv-cache/gpt_with_kv_cache.py", "label": "gpt_with_kv_cache.py", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!KVf4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3338a3b0-c3ad-4d37-9d3f-15db18db51ff_982x881.png", "label": "", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!qPKJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccfd3b14-df6e-4988-b986-851b4b207eae_1468x861.png", "label": "", "type": "a" }, { "href": "https://youtu.be/PetlIokI9Ao", "label": "Understanding PyTorch Buffers", "type": "a" }, { "href": "https://github.com/rasbt/LLMs-from-scratch/blob/main/ch04/03_kv-cache/gpt_with_kv_cache_optimized.py", "label": "gpt_with_kv_cache_optimized.py", "type": "a" }, { "href": "https://github.com/rasbt/LLMs-from-scratch/blob/main/pkg/llms_from_scratch/qwen3.py", "label": "qwen3.py", "type": "a" }, { "href": "https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/11_qwen3", "label": "README", "type": "a" }, { "href": "https://github.com/rasbt/LLMs-from-scratch/blob/main/pkg/llms_from_scratch/llama3.py", "label": "llama3.py", "type": "a" }, { "href": "https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/07_gpt_to_llama", "label": "README", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!-pE8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5d3a0bf-4a1a-439a-a83c-9046e1a57515_931x616.png", "label": "", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!O7SH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49ce3862-820f-4337-91b7-6b4603ec8f86_876x598.png", "label": "", "type": "a" }, { "href": "https://magazine.sebastianraschka.com/subscribe", "label": "paid subscription", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!woQp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png", "label": "", "type": "a" }, { "href": "https://www.amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167", "label": "brief review", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:47:24Z", "source_url": "https://magazine.sebastianraschka.com/p/coding-the-kv-cache-in-llms", "papers": [], "actual_video_files": [ { "local_path": "attachments/c9d8040bdb31/videos/PetlIokI9Ao.mp4", "bytes": 31648566, "valid": true, "height": 1080, "width": 1920, "duration": "812.061000", "status": "ok" } ], "body_chars": 20669, "expected_body_images": 16, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/c9d8040bdb31/pages/source.html", "local_html": "attachments/c9d8040bdb31/pages/article.html", "body_base_path": "attachments/c9d8040bdb31/pages/", "body_markdown_path": "attachments/c9d8040bdb31/pages/article.md" }