{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "Hugging Face Blog",
  "title_listed": "Continuous batching from first principles",
  "keywords": "连续批处理,推理,优化",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://huggingface.co/blog/continuous_batching",
  "publish_date": "2025-11",
  "corpus_stage": "推理部署",
  "notes": "连续批处理原理深度解析",
  "id": "efcaf252dca1",
  "collected_at": "2026-10-02T15:23:57",
  "title": "Continuous batching from first principles",
  "body_markdown": "Back to Articles\nContinuous batching\nPublished\n\t\t\t\t\tNovember 25, 2025\nUpdate on GitHub\nUpvote\n450\n+444\nRémi Ouazan Reboul\nror\nFollow\nArthur Zucker\nArthurZ\nFollow\nLuc Georges\nmcpotato\nFollow\nTL;DR: in this blog post, starting from attention mechanisms and KV caching, we derive continuous batching by optimizing for throughput.\nIf you've ever used Qwen, Claude, or any other AI chatbot, you've probably noticed something: it takes a while for the first word of the response to appear, and then words appear one-by-one on your screen with (hopefully) a regular and fast-paced frequency. That's because at the heart of it, all LLMs are just fancy next token predictors. An LLM first processes your entire prompt to produce one new token. Then it keeps adding tokens one by one, each time reading everything that came before, until it decides generation is over.\nThis generation process is computationally expensive: it requires passing the input through billions of parameters for each token generated. To make these models practical for real-world applications, particularly when serving many users simultaneously, researchers and engineers have developed a range of efficient inference techniques.\nOne of the most impactful optimizations is\ncontinuous batching\n, which attempts to maximize performance by processing multiple conversations in parallel and swapping them out when they are done.\nTo understand how continuous batching works and why it's so effective in high-load serving scenarios, we'll build up from the fundamentals of how LLMs process tokens.\nAttention\nThe attention mechanism is the central piece of how LLMs work. A language model processes text by breaking it down into pieces that we call tokens. We can conceptually think of \"tokens\" as \"words\", but sometimes a word might be composed of several tokens. For each token sequence, the network computes a prediction of what the next token should be.\nMany operations in the network are\ntoken-wise\n: each token is processed independently, and the output for a given token depends only on that token's content, not on any other tokens in the sequence. Operations like this include layer normalization or matrix multiplication. However, to create connections between words in a sentence, we need operations where tokens can influence each other.\nThis is where attention comes in.\nAttention layers are the only place where different tokens interact with each other\n. Understanding how a network connects tokens together means understanding attention.\nLet's see how this works in practice, in the case where there is only one input prompt.\nConsider the initial prompt\nI am sure this project\n, tokenized as 7 tokens:\n[<bos>, I, am, sure, this, pro, ject]\n. The\n<bos>\n, or \"Beginning of Sequence\", is a special token we add at the start of the prompt to tell the language model that a new conversation starts here.\nEach token is represented inside the network with a vector of length\nd\n(the\nhidden dimension\n). Therefore, the seven incoming tokens form a tensor\nx\nx\nx\nwith shape\n[\n1\n,\n7\n,\nd\n]\n\\left[1, 7, d \\right]\n[\n1\n,\n7\n,\nd\n]\n.\n1\nis the number of sequences, or batch size, which is just one in our case.\n7\nis the sequence length, and\nd\nis the hidden dimension, or the size of each token representation. Going forward, we'll use\nn\nn\nn\ninstead of\n7\nas the sequence length.\nInput tensor\nx\nx\nx\nis then projected by three matrices: the query projection\nW\nq\nW_q\nW\nq\n​\n,  the key projection\nW\nk\nW_k\nW\nk\n​\nand the value projection\nW\nv\nW_v\nW\nv\n​\n. This produces three tensors\nQ\nQ\nQ\n,\nK\nK\nK\nand\nV\nV\nV\n, all of shape\n[\n1\n,\nn\n,\nA\n]\n\\left[1, n , A \\right]\n[\n1\n,\nn\n,\nA\n]\n, where\nA\nA\nA\nis the dimension of the attention head. We call them the\nquery, key and value states,\nrespectively. This is represented on the left in the figure below.\nNext, tensors\nQ\nQ\nQ\nand\nK\nK\nK\nare multiplied together to measure similarity between tokens, producing a tensor of shape\n[\n1\n,\nn\n,\nn\n]\n\\left[ 1, n , n \\right]\n[\n1\n,\nn\n,\nn\n]\n. This is why we say that attention has quadratic complexity in sequence length. Computing\nQ\nK\nT\nQK^T\nQ\nK\nT\nrequires\nO\n(\nn\n2\nd\n)\n\\mathcal{O} \\left( n^2 d \\right)\nO\n(\nn\n2\nd\n)\noperations, so the cost is a square of\nn\nn\nn\nthe sequence length. It is represented on the right in the figure above.\nWe then apply a boolean\nattention mask\nto\nQ\nK\nT\nQK^T\nQ\nK\nT\nto control which tokens can interact, as represented in the figure below. In this figure, the attention mask is a\ncausal mask\n, meaning each token only interacts with tokens that came before it. This follows the intuition that a cause must come before its consequence, hence the name causal mask. The attention mask is crucial because it dictates all token interactions in the network.\nSet all attention mask values to False and no token will ever interact with another in the whole network.\nWe'll examine attention masks more closely in a few paragraphs.\nFinally, after applying the attention mask, we take a token-wise softmax (which is the same as saying a row-wise softmax) and multiply the result by the value projection\nV\nV\nV\nto get the output of one attention head, of shape\n[\n1\n,\nn\n,\nA\n]\n\\left[ 1, n , A \\right]\n[\n1\n,\nn\n,\nA\n]\n. We offer a visual summary of the whole process in the following figure.\nWe are going to use a lot of attention visualization in this post, so to simplify things, we are going to condense the figure above just a bit.\nWhy this matters:\nIn continuous batching,\nQ\nQ\nQ\n,\nK\nK\nK\n, and\nV\nV\nV\ncan have different numbers of tokens because, as we'll see, we'll be processing different stages (prefill and decode) at the same time. To make it more general, let's say\nQ\nQ\nQ\nhas shape\n[\n1\n,\nn\nQ\n,\nA\n]\n\\left[1, n_Q , A \\right]\n[\n1\n,\nn\nQ\n​\n,\nA\n]\n,\nK\nK\nK\nhas shape\n[\n1\n,\nn\nK\n,\nA\n]\n\\left[ 1, n_K , A \\right]\n[\n1\n,\nn\nK\n​\n,\nA\n]\n, and\nV\nV\nV\nhas shape\n[\n1\n,\nn\nV\n,\nA\n]\n\\left[ 1, n_V , A \\right]\n[\n1\n,\nn\nV\n​\n,\nA\n]\n.\nThe attention scores\nQ\nK\nT\nQK^T\nQ\nK\nT\nthen have shape\n[\n1\n,\nn\nQ\n,\nn\nK\n]\n\\left[ 1, n_Q , n_K \\right]\n[\n1\n,\nn\nQ\n​\n,\nn\nK\n​\n]\n, and the attention mask has the same shape since it's applied point-wise to the scores.\nAfter applying the attention mask and row-wise softmax, we multiply by\nV\nV\nV\n. Since we're multiplying a matrix of shape\n[\n1\n,\nn\nQ\n,\nn\nK\n]\n\\left[ 1, n_Q , n_K \\right]\n[\n1\n,\nn\nQ\n​\n,\nn\nK\n​\n]\nby one of shape\n[\n1\n,\nn\nV\n,\nA\n]\n\\left[ 1, n_V , A \\right]\n[\n1\n,\nn\nV\n​\n,\nA\n]\n, the inner dimensions must match:\nn\nK\n=\nn\nV\nn_K = n_V\nn\nK\n​\n=\nn\nV\n​\n. This means\nV\nV\nV\nand\nK\nK\nK\nalways have the same length, so we can simplify our visualizations by only showing\nK\nK\nK\n.\nDon't worry if this seems abstract: the figures will make it concrete.\nFurthermore, since we know that the attention mask is applied to\nQ\nK\nT\nQK^T\nQ\nK\nT\n, we know they have the same shape. Instead of representing the attention scores, we will represent the attention mask in its place.\nFinally, since\nQ\nQ\nQ\n,\nK\nK\nK\nand\nV\nV\nV\nare direct projections of\nx\nx\nx\n, no need to represent\nx\nx\nx\n. This gives the simplified figure where we only represent\nQ\nQ\nQ\n,\nK\nK\nK\nand the attention mask:\nThis representation also underlines how we can\nread an attention mask.\nWe read the mask row-by-row, which is the same as reading token-by-token: each row corresponds to one token's attention computation. A\ngreen square\nat position (row i, column j) means\nTrue\n: token j can influence token i. A\nwhite square\nmeans\nFalse\n: no interaction allowed.\nFor example, look at the third row for token \"\nam\n\". The \"\nI\n\" column is green, so \"\nI\n\" influences the computation of \"\nam\n\". The \"\npro\n\" column is white, so \"\npro\n\" doesn't influence \"\nam\n\" . This is causal masking at work: future tokens can't affect past ones.\nThe last layer of the model outputs a token prediction for each input token. In our context, generating the continuation of a single prompt, we only care about the next token prediction from the last token. The last token is \"\nject\n\" in the figure above, and the associated prediction is \"\nwill\n\".\nThe process we just described, where we take an entire input sequence, pass it through multiple attention layers and compute a score for the next token, is called\nprefill\n. This is because, as we'll see in a moment, much of the computation we performed can be cached and reused – hence, we are\nprefilling\nthe cache. Thanks to the use of this cache, sequence generation can proceed using much less compute in a phase called\ndecoding\n. In the decoding phase, generating one new token will be much faster than the initial full-sequence computation. Let's see why.\nTo continue generation, we begin a new forward pass, which would naively look like this:\nTo compute the attention scores of the new token, we still need the key and value projections of the previous tokens. So we need to repeat the matrix multiplication of the old tokens (in grey in the figure above) with\nW\nk\nW_k\nW\nk\n​\nand\nW\nv\nW_v\nW\nv\n​\nto retrieve a result that was already computed once before. In other terms, we are wasting compute. Let's see how we can avoid that.\nKV-cache\nRight off the bat, we notice that the last token does not impact the attention calculation of the other tokens:\nThis follows the idea of the causal mask: since \"\nwill\n\" comes after all previous tokens, it does not change their attention calculation.\nFor text generation, causal attention is by far the most common, so we will focus on that case from now on. Keep in mind that non-causal attention schemes can also be used, especially when dealing with images.\nConsidering we only need the next-token prediction for the \"\nwill\n\" token, we can simplify the attention mechanism by only computing the output for this token.\nMoreover, we already computed the\nK\nK\nK\nand\nV\nV\nV\nstates for the tokens \"\n<bos>\n\", … ,  \"\nject\n\" during the previous forward pass: if they have been stored, we do not need to recompute them again. This is the\nKV cache\n: the list of key and value states created during generation. It essentially allows one to reduce the compute cost of generating token\nn\n+\n1\nn+1\nn\n+\n1\nfrom\nO\n(\nn\n2\n)\n\\mathcal{O} \\left( n^2 \\right)\nO\n(\nn\n2\n)\nto\nO\n(\nn\n)\n\\mathcal{O} \\left( n \\right)\nO\n(\nn\n)\nby avoiding recomputation of key and value projections, while paying a memory cost of\nO\n(\nn\n)\n\\mathcal{O} \\left( n \\right)\nO\n(\nn\n)\n.\nIn the figure above, only the tokens in white are computed: instead of computing the keys and values for 8 tokens, we compute them for 1. You can see that through KV caching, a lot of compute is saved.\nYou can check\nthis post\nfor more visualizations of KV caching, or\nthis one\nfor a practical implementation example.\nLet's be a bit more specific about the cache size, because it's a good opportunity to examine the shapes present in our model. For a model with\nL\n\\mathcal L\nL\nattention layers and\nH\nH\nH\nattention heads with head dimension\nA\nA\nA\n, the total cache size needed to store one token will be\n2\n∗\nL\n∗\nA\nH\n2 *\\mathcal L * AH\n2\n∗\nL\n∗\nA\nH\nwith a factor of\n2\n2\n2\nto account for both\nK\nK\nK\nand\nV\nV\nV\n.\nFor instance, Llama-2-7B with\nL\n=\n32\n\\mathcal{L}=32\nL\n=\n32\nlayers,\nH\n=\n32\nH=32\nH\n=\n32\nheads, and\nA\n=\n128\nA=128\nA\n=\n128\nrequires\n2\n×\n32\n×\n128\n=\n8\n,\n192\n2 \\times 32 \\times 128 = 8,192\n2\n×\n32\n×\n128\n=\n8\n,\n192\nvalues per token per layer. With\nfloat16\nprecision, this takes\n2\nA\nH\n×\n2\n2AH \\times 2\n2\nA\nH\n×\n2\nbytes\n=\n16\n= 16\n=\n16\nKB in memory.\nKV caching is useful when we want to generate the next token, which is a stage we call\ndecoding\n. But it can also be useful in the\nprefill\nstage, when we process the initial prompt and have many input tokens. Especially when there are large initial prompts that don't fit in GPU memory all at once.\nChunked prefill\nUp till now, we have looked at an example of prefill where we have\nn\n=\n7\nn=7\nn\n=\n7\ntokens, but in practice initial prompts can be much longer. For instance, when using Cursor, you can add your repository to the prompt, where it acts as context: this significantly increases the prompt size. In such cases, the memory needed to store the activations for\nn\nn\nn\ntokens can be larger than the available memory on the GPU. Thus we cannot perform prefill in a single forward pass: we have to split the prefill in chunks. This is called\nchunked prefill\n, and it's going to be one of the components needed to enable efficient inference.\nLet's pretend that the available memory is very constrained, and that we can only pass\nm\n=\n4\nm=4\nm\n=\n4\ntokens per forward pass. If we have an initial prompt with\nn\n=\n7\nn = 7\nn\n=\n7\ntokens, we need to split it in\n⌈\nn\n/\nm\n⌉\n=\n2\n\\lceil n /m \\rceil = 2\n⌈\nn\n/\nm\n⌉\n=\n2\nchunks (rounding up 7/4 = 1.75 to 2). We illustrate the example below using the same\nn\nn\nn\nand\nm\nm\nm\nnotations:\nWe can do that thanks to the KV cache. We store the KV states during the first prefill split, and during the second prefill split, we prepend the stored KV states to the new KV states. We also adapt the attention mask accordingly. Visually, it looks like we split the non-chunked prefill in the middle.\nThe key insight: cached KV states let us process the prompt incrementally without losing information.\nAlthough we showed here an example where we split the prefill into 2 chunks, chunked prefill can be used to split the prefill in any way we want, adapting flexibly to memory constraints.\nWe are now finally equipped with all the tools we need to understand Continuous Batching.\nContinuous batching\nIn our previous examples we have only considered the case of batch size one, i.e. we only generate tokens for one prompt at a time. In the context of evaluation or model serving, we want to generate tokens for a large number of prompts. To increase the\nthroughput\n, which is the number of tokens generated per second, the best course of action is to generate tokens in parallel for a batch of several prompts.\nTo batch prompts together, the naive way is to add an axis to both input tensors: token sequence and attention mask. However, this comes with a constraint on the shape of the inputs: we need all prompts to have the same length, because tensors must be rectangular. To achieve this, we usually add padding on the left so the new token prediction always comes from the rightmost token. We also modify the attention mask of each prompt accordingly, as shown below:\nwhere the padding tokens\n<pad>\nare coloured in orange. Then we can perform the forward pass as we used to, with the added dimension of the batch size. This is called\nbatched generation\n: efficient for same-length prompts, but wasteful when lengths vary.\nIt is illustrated below, through 4 steps of generation: one prefilling step (at the top) and 3 decoding steps (below each \"Forward pass\" lines).\nwhere\n<eos>\nmeans \"End Of Sequence\", this is a special token to indicate the model has reached the end of generation for the corresponding sequence.\nThe drawback of batched generation is that if one prompt finishes generation before the other one by generating an\n<eos>\ntoken, all further generated tokens are useless. And this goes on until the longest request of the batch finishes. Of course, we can remove the prompts that have reached an\n<eos>\ntoken from the batch and save some compute and memory, but saving resources is not the goal here: throughput is.\nInstead of just removing the finished prompt from the batch, we can replace it with a prompt that's waiting for generation. We will call this\ndynamic scheduling\n, or dynamic batching. Dynamic scheduling is great to maintain throughput while ensuring any token generated by a forward pass is relevant. But because of the way we batched prompts together, it has a major drawback: we need a lot of padding when swapping prompts. That's because the newly-inserted prompt needs to go through prefill while the other prompts are decoding one token at a time. So there is almost as much padding as there are tokens in the newly-inserted prompt.\nThe problem becomes even worse when batch size increases and initial prompts are long. The padding cost grows quadratically with both batch size and prompt length. If we have a batch of\nB\nB\nB\nprompts that are in decoding phase and one finishes, dynamically introducing a prompt of\nn\nn\nn\ninitial tokens in the batch requires\n(\nn\n−\n1\n)\n(\nB\n−\n1\n)\n(n-1)(B-1)\n(\nn\n−\n1\n)\n(\nB\n−\n1\n)\npadding tokens. For instance, with\nB\n=\n8\nB=8\nB\n=\n8\nand\nn\n=\n100\nn=100\nn\n=\n100\n, we'd need\n99\n×\n7\n=\n693\n99 \\times 7 = 693\n99\n×\n7\n=\n693\npadding tokens!\nFurthermore, practical optimizations like CUDA graphs or\ntorch.compile\nrequire static tensor shapes. This forces us to pad all prompts to a fixed maximum length, dramatically increasing the padding waste.\nAt this point, our main problem is padding, which is a consequence of the axis we added to batch sentences together. Thus, the ideal would be to get rid of this axis entirely, a radical rethinking of batching. If we do so, the only way to batch prompts together is to concatenate them:\nBut we don't want tokens from prompt 0 to interact with the tokens of prompt 1! Luckily for us, we have a way to control how tokens interact with one another: the attention mask. How we do this is displayed below:\nAlthough we use different tints of green to illustrate the different parts of the attention mask, this is still a boolean mask with only greens for\nTrue\nand white for\nFalse\n. \nThis way of batching prompts together is called\nragged batching\n(because sequence lengths are 'ragged' or uneven), and it offers the benefit of added throughput without introducing the need for padding tokens.\nIn the figure above, we use ragged batching to combine two full prompts together, but we can batch as many as memory allows. The only limit is\nm\nm\nm\n, the number of tokens we can fit in a batch, with\nm\nm\nm\ndepending on the available memory on the GPU.\nRagged batching is one of the key components of continuous batching. To maximize throughput, we can combine prefill and decoding sequences following an algorithm like this:\nWe try to always reach our memory budget of\nm\nm\nm\ntokens per batch\nWe first add all the prompts in decoding phase to the batch, each accounting for 1 token\nWe fill the remaining space with prefill phase prompts, relying on the flexibility of chunked prefill to split inputs as needed\nDynamic scheduling is the final piece that contributes to the\ncontinuous batching\ntechnique: we remove finished prompts from the batch as soon as they are done, and replace them with new chunked prompts that correspond to incoming requests.\nThis combination of ragged batching and dynamic scheduling is called\ncontinuous batching\n, and it's the technique that powers modern LLM serving systems.\nConclusion\nContinuous batching combines three key techniques to maximize throughput in LLM serving:\nKV caching\nto avoid recomputing past token representations\nChunked prefill\nto handle variable-length prompts within memory constraints\nRagged batching with dynamic scheduling\nto eliminate padding waste and keep the GPU fully utilized\nBy removing the batch dimension and using attention masks to control token interactions, continuous batching allows mixing prefill and decode phases in the same batch, dramatically improving efficiency for serving multiple requests. This is why services like ChatGPT can handle thousands of concurrent users efficiently.\nIn the next article in this series, we make continuous batching much more efficient by introducing\nasynchronous batching\n.\nRead it here!\nIf you'd like to see a deep dive on other continuous batching topics, please let us know in the comments!\nAcknowledgement: thanks to Arthur Zucker for producing the initial concept for the figures used in this article. And thanks to Arthur Zucker, Luc Georges, Lysandre Debut, Merve Noyan and Pedro Cuenca for all providing helpful reviews.\nMore Articles from our Blog\ntransformers\npytorch\noptimization\nUnlocking asynchronicity in continuous batching\n68\nMay 14, 2026\nvllm\ntransformers\ninference\nNative-speed vLLM transformers modeling backend\n77\nJuly 8, 2026\nCommunity\nrupanshu\nNov 26, 2025\nThanks for the article! Very helpful. One question though, in the last image, just before conclusion, shouldn't there be a\n<bos>\ntoken?\nSee translation\n4 replies\n·\nArthurZ\nArticle author\nNov 26, 2025\nYes!\n❤️\n2\n2\n+\nExpand 3\n\t\t\t\t\t\treplies\nbg90\nNov 26, 2025\nNice reference to John Cena! 👋\nSee translation\n2 replies\n·\nmcpotato\nArticle author\nNov 27, 2025\nwhat reference?\n👀\n5\n5\n+\nExpand 1\n\t\t\t\t\t\treply\nnaman5a\nNov 26, 2025\nLove this article :)\n@\nArthurZ\nSee translation\n❤️\n1\n1\n+\nReply\nsrinivasbilla\nNov 27, 2025\nThis is amazing! I know I'm probably being greedy but code examples would be a great addition. In fact a \"vllm from scratch\" series would go crazy\nSee translation\n❤️\n7\n7\n➕\n5\n5\n+\nReply\nivanfioravanti\nNov 27, 2025\nAmazing article! Thanks for it 🙏\nSee translation\n❤️\n1\n1\n+\nReply\nModfiededition\nNov 29, 2025\nDiagrams have been very helpful in understanding the concepts. Excited for the next articles in the series :)\nSee translation\n❤️\n1\n1\n+\nReply\nknef2\nNov 29, 2025\nLove the breakdown!\n❤️\n1\n1\n+\nReply\nretry51776\nDec 6, 2025\nGreat read! Thanks\nReply\nalbertmichaelj\nDec 11, 2025\nI love these kinds of articles. They are the primary reason that I follow the Huggingface blog! Please continue the series (and this style of blog posts).\nSee translation\n❤️\n4\n4\n+\nReply\nlwtwl\nDec 26, 2025\nIs this diagram drawn incorrectly? Should the part I highlighted in the green box be in the first two columns on the left?\nSee translation\n1 reply\n·\nakash-inferq\nMar 12\ntechnically it is representing the padding unused/wasted compute so even if it being on the left would actually be the same thing/correct representation but I think he wants to show that that is the actual wasted compute (to be quantifiable)\nSee translation\naarhi-mahanta\nJan 14\nWhat an amazing explanation! The visuals are much helpful!\nSee translation\nReply\nqdai1\nFeb 3\nGreat work! When will the next article about paged attention be released?\nSee translation\nReply\nzinchse\nMay 3\nThanks for the article! I'd like to add one note: the primary reason we need chunked prefill isn't OOM\nisn't memory constraint\n(otherwise we could simply solve it with Flash Attention). The more practical reason is that a long prefill blocks all decode requests in the batch. Chunking it lets decode requests make progress between chunks. Moreover, since prefill chunks make the forward pass compute-bound, decode tokens can piggyback on the same batch at nearly zero marginal cost - they're just extra rows in an already large matmul. This turns memory-bound decode-only batches into better-utilized compute-saturating ones. See the SARATHI paper for details.\nSee translation\n👍\n1\n1\n+\nReply\n007superher\nMay 14\nNice read. Thanks a lot!\nSee translation\nReply\nkyars\nJun 11\nreally impressive!\nReply\nethanwu97\nAug 18\nThere seems to be a minor mistake in the \"Chunked Prefill\" illustration - was the token \"that\" accidentally added?\nSee translation\nReply\nEdit\nPreview\nUpload images, audio, and videos by dragging in the text input, pasting, or\nclicking here\n.\nTap or paste here to upload images\nComment\n·\nSign up\nor\nlog in\nto comment\nUpvote\n450\n+438",
  "metadata": {
    "title": "Continuous batching from first principles",
    "description": "We’re on a journey to advance and democratize artificial intelligence through open source and open science.",
    "keywords": null,
    "author": null,
    "og:title": "Continuous batching from first principles",
    "og:description": "We’re on a journey to advance and democratize artificial intelligence through open source and open science.",
    "og:type": "website",
    "og:url": "https://huggingface.co/blog/continuous_batching",
    "og:image": "https://huggingface.co/blog/assets/continuous_batching/thumbnail.png",
    "twitter:card": "summary_large_image",
    "twitter:site": "@huggingface",
    "twitter:image": "https://huggingface.co/blog/assets/continuous_batching/thumbnail.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/850de715ec67435ba6b53d942719af9d.svg",
      "bytes": 1282,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-9b513b74ca0c4e4ef4d0.svg",
      "src": "https://huggingface.co/avatars/850de715ec67435ba6b53d942719af9d.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/52dc57576e7cd3945c55cece1cf87969.svg",
      "bytes": 1282,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-0b1bcddc061ca162a019.svg",
      "src": "https://huggingface.co/avatars/52dc57576e7cd3945c55cece1cf87969.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Tux8SGRpro7_OesS7VJrw.png",
      "bytes": 16232,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-bf2bd7ac25a3c38b1726.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Tux8SGRpro7_OesS7VJrw.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/655eeb5532537bcc8d7460ab/gV_GfYq-GEyi1cbCTQe0r.jpeg",
      "bytes": 14410,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-194ec7d9d6c3803d5b39.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/655eeb5532537bcc8d7460ab/gV_GfYq-GEyi1cbCTQe0r.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/b4260b60fece4fadb13ea05fb18cd6c5.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-5199d2af16beabf4ae17.svg",
      "src": "https://huggingface.co/avatars/b4260b60fece4fadb13ea05fb18cd6c5.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/e1f19a99964c61a3b2f9318a12072981.svg",
      "bytes": 1283,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-56b36a4b2f6dc9e3f269.svg",
      "src": "https://huggingface.co/avatars/e1f19a99964c61a3b2f9318a12072981.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/banner.png",
      "bytes": 505776,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/003.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/banner.png",
      "alt": "Title card",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/proj_and_mul.png",
      "bytes": 204017,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/004.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/proj_and_mul.png",
      "alt": "proj_and_mul.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/masking_and_softmax.png",
      "bytes": 164656,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/005.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/masking_and_softmax.png",
      "alt": "masking_and_softmax.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/attention.png",
      "bytes": 262484,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/006.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/attention.png",
      "alt": "attention.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/simple_attention.png",
      "bytes": 63973,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/007.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/simple_attention.png",
      "alt": "simple_attention.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/naive_generate.png",
      "bytes": 75161,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/008.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/naive_generate.png",
      "alt": "naive_generate.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/cant_see_me.png",
      "bytes": 104737,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/009.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/cant_see_me.png",
      "alt": "cant_see_me.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/kv_cache.png",
      "bytes": 35735,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/010.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/kv_cache.png",
      "alt": "kv_cache.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/chunked_prefill.png",
      "bytes": 164366,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/011.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/chunked_prefill.png",
      "alt": "chunked prefill.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/padding.png",
      "bytes": 107919,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/012.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/padding.png",
      "alt": "padding.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/batched_generation.png",
      "bytes": 169886,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/013.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/batched_generation.png",
      "alt": "batched_generation.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/dynamic_batching.png",
      "bytes": 140782,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/014.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/dynamic_batching.png",
      "alt": "dynamic_batching.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/concatenate.png",
      "bytes": 40972,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/015.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/concatenate.png",
      "alt": "concatenate.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/ragged_batching.png",
      "bytes": 169193,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/016.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/ragged_batching.png",
      "alt": "ragged_batching.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/continuous_batching.png",
      "bytes": 229856,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/017.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/continuous_batching.png",
      "alt": "continuous_batching.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/blog/assets/continuous_async/thumbnail.png",
      "bytes": 2433012,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-60c6d213869210179d0b.png",
      "src": "https://huggingface.co/blog/assets/continuous_async/thumbnail.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/6123945a0ed258ebc83f3d56/8wMHFQHEV24G_ljl4kPxQ.jpeg",
      "bytes": 9028,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/000.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/6123945a0ed258ebc83f3d56/8wMHFQHEV24G_ljl4kPxQ.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1617264212503-603d25b75f9d390ab190b777.jpeg",
      "bytes": 9492,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-75a62629485c7cda2c4d.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1617264212503-603d25b75f9d390ab190b777.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/608aabf24955d2bfc3cd99c6/-YxmtpzEmf3NKOTktODRP.jpeg",
      "bytes": 9712,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-d4a5f1f6ae045b66663f.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/608aabf24955d2bfc3cd99c6/-YxmtpzEmf3NKOTktODRP.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/blog/assets/native-speed-vllm-transformers-backend/thumbnail.png",
      "bytes": 6259641,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-9de8868c9fd090f0315d.png",
      "src": "https://huggingface.co/blog/assets/native-speed-vllm-transformers-backend/thumbnail.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ov-55uEKP4mJhavhdQ42x.png",
      "bytes": 6084,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-18d6cc7e6086dfe62195.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ov-55uEKP4mJhavhdQ42x.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/5e3aec01f55e2b62848a5217/PMKS0NNB4MJQlTSFzh918.jpeg",
      "bytes": 11598,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-741cc42f07a702f49510.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/5e3aec01f55e2b62848a5217/PMKS0NNB4MJQlTSFzh918.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/eb2ccce59bc69345f9a4a32fb37a5bd9.svg",
      "bytes": 1280,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-8e78a6061248a8e3bd4b.svg",
      "src": "https://huggingface.co/avatars/eb2ccce59bc69345f9a4a32fb37a5bd9.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-uploads.huggingface.co/production/uploads/659f915228725b8dad817742/sS2g0VXAgnX5zUDnGlYkY.png",
      "bytes": 135866,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/018.png",
      "src": "https://cdn-uploads.huggingface.co/production/uploads/659f915228725b8dad817742/sS2g0VXAgnX5zUDnGlYkY.png",
      "alt": "image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1674683851722-62441cb7456803e95009a08f.jpeg",
      "bytes": 17020,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/001.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1674683851722-62441cb7456803e95009a08f.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/339b70298fe44c943be24dc39ada0e52.svg",
      "bytes": 1284,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-91b2b865783f9ede742c.svg",
      "src": "https://huggingface.co/avatars/339b70298fe44c943be24dc39ada0e52.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VFO_-tSfESObMKW9jPv5_.png",
      "bytes": 6210,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-a58b8243d2c21e5bfdfa.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VFO_-tSfESObMKW9jPv5_.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/abab3a7115123010c728a17203eb6aa8.svg",
      "bytes": 1282,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-8e8ddc5240b7fa46d486.svg",
      "src": "https://huggingface.co/avatars/abab3a7115123010c728a17203eb6aa8.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1666977434736-617bc8d1000dbbbf7c225eed.png",
      "bytes": 24660,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/002.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1666977434736-617bc8d1000dbbbf7c225eed.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/7BnCVOvg9hY2EXvg3h4p6.png",
      "bytes": 7670,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-a4b289ec2bab12b68bc5.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/7BnCVOvg9hY2EXvg3h4p6.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/63b97e5eefe99543b34c7dd6/lRnA9OerlezpA5rQo9rJO.jpeg",
      "bytes": 11048,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-465e42797765cbc323ce.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/63b97e5eefe99543b34c7dd6/lRnA9OerlezpA5rQo9rJO.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/64345a5f4b34368fdb045ef4/hll9uuO_67PETaYAgQGUQ.jpeg",
      "bytes": 13132,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-c4e774a3dd6e8fa41c58.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/64345a5f4b34368fdb045ef4/hll9uuO_67PETaYAgQGUQ.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/a6736ae2e8afa41e71d63ef519516f17.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-35461983d5f08654a1a0.svg",
      "src": "https://huggingface.co/avatars/a6736ae2e8afa41e71d63ef519516f17.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/95a14d21ccba58b70da6eaa65d396753.svg",
      "bytes": 1282,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-d52675e4e4abec0e56df.svg",
      "src": "https://huggingface.co/avatars/95a14d21ccba58b70da6eaa65d396753.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/ed8eb5b07702fbd9ec34938c4712421a.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-f65502d6803bc938e0bf.svg",
      "src": "https://huggingface.co/avatars/ed8eb5b07702fbd9ec34938c4712421a.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/61ccc355c66a11221454d983895d9646.svg",
      "bytes": 1284,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-b704a6c1c7f9c4f8431a.svg",
      "src": "https://huggingface.co/avatars/61ccc355c66a11221454d983895d9646.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/3fa10fde5147bf00724e3e1453006512.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-9ce552827b5dd7525ec1.svg",
      "src": "https://huggingface.co/avatars/3fa10fde5147bf00724e3e1453006512.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-uploads.huggingface.co/production/uploads/627b1f26772b403388b9c5ec/z85mqlbCm7xfkhETvfk8N.png",
      "bytes": 232587,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/019.png",
      "src": "https://cdn-uploads.huggingface.co/production/uploads/627b1f26772b403388b9c5ec/z85mqlbCm7xfkhETvfk8N.png",
      "alt": "image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/35d7f8e04a873d2aca4470b5d811bb24.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-fab98587f72b0c5c1840.svg",
      "src": "https://huggingface.co/avatars/35d7f8e04a873d2aca4470b5d811bb24.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/5bdd5b2dcae65e17cbcbef3ec4b7e03f.svg",
      "bytes": 1280,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-ce2347700777e3a1be91.svg",
      "src": "https://huggingface.co/avatars/5bdd5b2dcae65e17cbcbef3ec4b7e03f.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/662450b21a2471de2510e44201f4627f.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-8081324895195718a7ab.svg",
      "src": "https://huggingface.co/avatars/662450b21a2471de2510e44201f4627f.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ecpx2wUchQzthZFB14Hcp.jpeg",
      "bytes": 5602,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-b998753dbdfba09e5888.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ecpx2wUchQzthZFB14Hcp.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/3ec6e90c1abfbedd86985ccba3a56cb5.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-67e782653520ae778f9b.svg",
      "src": "https://huggingface.co/avatars/3ec6e90c1abfbedd86985ccba3a56cb5.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/JlLx6zb-QcMSyfOhz_QmZ.png",
      "bytes": 7952,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-d20e74efa9c5c14d207b.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/JlLx6zb-QcMSyfOhz_QmZ.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/48aa54fc9cd1858e80c2ed82e9716c8d.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-0832148c70ad2ad6f9cc.svg",
      "src": "https://huggingface.co/avatars/48aa54fc9cd1858e80c2ed82e9716c8d.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-uploads.huggingface.co/production/uploads/6a7ba2f0092e18bf18bce88c/I4Rnw5wY5LJe357lkmUGe.png",
      "bytes": 382090,
      "reused": true,
      "local_path": "attachments/efcaf252dca1/images/020.png",
      "src": "https://cdn-uploads.huggingface.co/production/uploads/6a7ba2f0092e18bf18bce88c/I4Rnw5wY5LJe357lkmUGe.png",
      "alt": "Screenshot 2026-08-17 at 8.54.01 PM",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/cff2eb4d9615184a7978f0c9213a59e6.svg",
      "bytes": 1284,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-ca0dae7e6e1f419f0bd1.svg",
      "src": "https://huggingface.co/avatars/cff2eb4d9615184a7978f0c9213a59e6.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/a739de0b362d95c0d2fa46e5c35d54b5.svg",
      "bytes": 1282,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-d5bb11aa8225bfc72ed2.svg",
      "src": "https://huggingface.co/avatars/a739de0b362d95c0d2fa46e5c35d54b5.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/e89c521885f73565503e650814fc6d7f.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-52d760946e4b5c74da93.svg",
      "src": "https://huggingface.co/avatars/e89c521885f73565503e650814fc6d7f.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/67fcc54b597b78184d77087d/x2fViSuvM2bQTb0FYijLU.png",
      "bytes": 5470,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-a4b56926a389ccc91687.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/67fcc54b597b78184d77087d/x2fViSuvM2bQTb0FYijLU.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/671f04868edebd4c93fb40c7fb24b48e.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/efcaf252dca1/images/body-5c28d3df0226bf81567f.svg",
      "src": "https://huggingface.co/avatars/671f04868edebd4c93fb40c7fb24b48e.svg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://huggingface.co/blog",
      "label": "Back to Articles",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/blog/blob/main/continuous_batching.md",
      "label": "Update on GitHub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/login?next=%2Fblog%2Fcontinuous_batching",
      "label": "Upvote 450",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/anushkadhiman",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/CoolCarlos",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/fuzzycoder3794",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/whybe-choi",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/rfs",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/ashfaqr",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/ror",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/ArthurZ",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/mcpotato",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/banner.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/proj_and_mul.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/masking_and_softmax.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/attention.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/simple_attention.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/naive_generate.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/cant_see_me.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/kv_cache.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/not-lain/kv-caching",
      "label": "this post",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/kv-cache",
      "label": "this one",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/chunked_prefill.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/padding.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/batched_generation.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/dynamic_batching.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/concatenate.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/ragged_batching.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/continuous_batching/continuous_batching.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/continuous_async",
      "label": "Read it here!",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/native-speed-vllm-transformers-backend",
      "label": "vllm transformers inference Native-speed vLLM transformers modeling backend 77 July 8, 2026",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/rupanshu",
      "label": "rupanshu",
      "type": "a"
    },
    {
      "href": "https://cdn-uploads.huggingface.co/production/uploads/659f915228725b8dad817742/sS2g0VXAgnX5zUDnGlYkY.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/kylehh",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/liam255",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/bg90",
      "label": "bg90",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/naman5a",
      "label": "naman5a",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/srinivasbilla",
      "label": "srinivasbilla",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/ivanfioravanti",
      "label": "ivanfioravanti",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/Modfiededition",
      "label": "Modfiededition",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/knef2",
      "label": "knef2",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/retry51776",
      "label": "retry51776",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/albertmichaelj",
      "label": "albertmichaelj",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/lwtwl",
      "label": "lwtwl",
      "type": "a"
    },
    {
      "href": "https://cdn-uploads.huggingface.co/production/uploads/627b1f26772b403388b9c5ec/z85mqlbCm7xfkhETvfk8N.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/akash-inferq",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/aarhi-mahanta",
      "label": "aarhi-mahanta",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/qdai1",
      "label": "qdai1",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/zinchse",
      "label": "zinchse",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/007superher",
      "label": "007superher",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/kyars",
      "label": "kyars",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/ethanwu97",
      "label": "ethanwu97",
      "type": "a"
    },
    {
      "href": "https://cdn-uploads.huggingface.co/production/uploads/6a7ba2f0092e18bf18bce88c/I4Rnw5wY5LJe357lkmUGe.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/join?next=%2Fblog%2Fcontinuous_batching",
      "label": "Sign up",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/mohitlenka10",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/sahilpereira",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/belingbeling",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/zikunchen",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/zehao233",
      "label": "",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:26Z",
  "source_url": "https://huggingface.co/blog/continuous_batching",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 22877,
  "expected_body_images": 57,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/efcaf252dca1/pages/source.html",
  "local_html": "attachments/efcaf252dca1/pages/article.html",
  "body_base_path": "attachments/efcaf252dca1/pages/",
  "body_markdown_path": "attachments/efcaf252dca1/pages/article.md"
}