{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Aleksei Romanov",
  "title_listed": "High-Throughput LLM Inference & Training: A Deep Dive into vLLM",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/g_factor/high-throughput-llm-inference-training-a-deep-dive-into-vllm-5818",
  "publish_date": "2026-09-21",
  "corpus_stage": "",
  "notes": "",
  "id": "81b67fe28cdb",
  "collected_at": "2026-10-02T13:54:45",
  "title": "High-Throughput LLM Inference & Training: A Deep Dive into vLLM - DEV Community",
  "body_markdown": "Editor's Note:\nOriginally published on the\ng factor engineering blog\n. All benchmarks and telemetry in this article were conducted on dedicated NVIDIA H100 and H200 clusters on\ngft-studio\n.\nIf you have ever stared at\nnvidia-smi\nduring a production inference run and felt your heart sink seeing 12% GPU compute utilization while users complained about sluggish generation, you have run headfirst into the central reality of modern LLMs:\ntext generation is a memory bandwidth problem disguised as a compute problem\n.\nWhen a transformer generates text token-by-token, your multi-thousand-dollar GPU spends almost none of its time flexing its tensor cores. Instead, it spends virtually all its time acting like a high-speed forklift in a warehouse—shuffling tens of gigabytes of model weights and historical Key-Value (KV) cache tensors back and forth across High Bandwidth Memory (HBM) for every single emitted word.\nThe moment you push beyond single-user toy demos into multi-tenant production or high-throughput reinforcement learning (RL) rollouts, naive PyTorch stacks hit a wall: memory fragmentation chews up your VRAM, static batching leaves GPUs idling in massive \"bubbles,\" and host-side driver overhead leaves silicon starving. That is why vLLM took over the inference world—not through arcane black magic, but through elegant, battle-tested operating systems engineering:\nPagedAttention\nand\ncontinuous iteration-level batching\n.\nKey Concepts at a Glance\nToken:\nA unit of model input or output, typically corresponding to 3–4 characters of English text.\nPrefill phase:\nProcessing the prompt in parallel—dense compute-heavy matrix multiplication (GEMM).\nDecode phase:\nGenerating subsequent tokens sequentially—memory-bandwidth-bound matrix-vector operations (GEMV).\nKV Cache:\nVRAM tensors storing past Key and Value representations so attention doesn't recompute historical tokens.\nContinuous batching:\nIteration-level scheduling that admits waiting requests immediately when any sequence finishes.\n1. The KV Cache Bottleneck: Why Memory Bites Back\nAutoregressive transformers do not generate a paragraph all at once. When a prompt arrives, the model processes all input tokens in parallel during the\nprefill phase\n. This is dense, compute-heavy matrix multiplication (GEMM)—the kind of workload GPUs were born to do.\nBut the moment generation begins (the\ndecode phase\n), everything changes. To generate token #501, the model needs self-attention over the preceding 500 tokens. Recalculating all 500 token representations from scratch on every single step would be an\nO\n(\nN\n2\n)\nO(N^2)\nO\n(\nN\n2\n)\ncomputational nightmare. So, we cache the intermediate Key and Value vector representations in GPU VRAM:\nM\nKV\n=\n2\n×\nL\n×\nH\nKV\n×\nd\nhead\n×\nT\n×\nb\nM_{\\text{KV}} = 2 \\times L \\times H_{\\text{KV}} \\times d_{\\text{head}} \\times T \\times b\nM\nKV\n​\n=\n2\n×\nL\n×\nH\nKV\n​\n×\nd\nhead\n​\n×\nT\n×\nb\nThe first factor of 2 accounts for both Keys and Values.\nL\nL\nL\nis the number of full-attention layers,\nH\nKV\nH_{\\text{KV}}\nH\nKV\n​\nis the number of KV heads per layer,\nd\nhead\nd_{\\text{head}}\nd\nhead\n​\nis the head dimension,\nT\nT\nT\nis the active sequence token length, and\nb\nb\nb\nis the precision in bytes (e.g., 2 bytes for BF16).\nCrucially, modern open-weight architectures (like\nQwen3.6-27B\n) use Grouped-Query Attention (GQA) to keep memory sane: 16 full-attention layers with 4 KV heads and a head dimension of 256. At an 8,192-token sequence in BF16, that single sequence commands:\nM\nKV\n=\n2\n×\n16\n×\n4\n×\n256\n×\n8\n,\n192\n×\n2\nbytes\n=\n536\n,\n870\n,\n912\nbytes\n≈\n0.54\nGiB\nM_{\\text{KV}} = 2 \\times 16 \\times 4 \\times 256 \\times 8{,}192 \\times 2\\text{ bytes} = 536{,}870{,}912\\text{ bytes} \\approx 0.54\\text{ GiB}\nM\nKV\n​\n=\n2\n×\n16\n×\n4\n×\n256\n×\n8\n,\n192\n×\n2\nbytes\n=\n536\n,\n870\n,\n912\nbytes\n≈\n0.54\nGiB\nHalf a gigabyte sounds manageable—until you realize what happens under load. In standard PyTorch or basic Hugging Face\ngenerate()\npipelines, memory management is primitive. Dynamic containers like\nDynamicCache\nallocate buffers on the fly. In a multi-user service where one user asks for a 20-line bash script and another submits a 6,000-token legal document, allocating and reallocating memory turns your GPU VRAM into Swiss cheese.\nThis is\nexternal memory fragmentation\n: you might have 15 GB of total free VRAM reported, but because it is shattered into non-contiguous fragments, the next request asking for a contiguous 2 GB block crashes with a catastrophic\nCUDA Out of Memory\n(OOM).\n2. PagedAttention: Borrowing a 50-Year-Old OS Masterpiece\nBack in the 1960s, operating system pioneers realized that requiring programs to live in contiguous physical RAM was madness. Their solution was virtual memory paging: chop memory into fixed pages (usually 4 KB) and let the hardware map arbitrary virtual addresses to scattered physical pages via a page table.\nvLLM brought this exact insight to GPU memory with\nPagedAttention\n. Instead of reserving a giant contiguous chunk of VRAM for each sequence's worst-case length, it chops the KV cache into fixed-size physical blocks (typically holding 16 or 32 tokens).\nVisualizing the KV Cache Reservation\nBelow is a comparison of memory allocation policies when serving 3 concurrent requests (holding 3, 5, and 2 tokens respectively) with a maximum length of 12 tokens:\nFigure 1: Contiguous reservation baseline — Each request reserves room for 12 tokens: 36 slots reserved, 10 used, 26 unused (72% memory wasted).\nFigure 2: Paged block allocation — Four-token blocks reserve 4 + 8 + 4 = 16 slots: 10 used, only 6 unused. Blocks live anywhere in physical memory, freeing 20 slots for other user requests.\nA centralized\nBlock Table\nmaps logical token positions to physical blocks:\nOn-Demand Allocation:\nBlocks are handed out only when tokens are actually generated. Only the very last block of a running sequence has any unused slots.\nSlashing Memory Waste to < 4%:\nInternal fragmentation virtually disappears. Because you aren't hoarding empty memory buffers for worst-case prompts, the exact same GPU hardware can suddenly host\n2x to 4x more concurrent user streams\nwithout breaking a sweat.\nCopy-on-Write (CoW) Branching:\nThis is a game-changer for reinforcement learning (RL) and parallel tree search. In algorithms like GRPO, the model generates 8 or 16 candidate rollouts from the exact same prompt. With PagedAttention, all 16 candidate rollouts physically share the prompt's KV memory pages. Physical memory is cloned only when individual candidate completions diverge.\n3. Continuous Batching: Ending the Tyranny of the Slowest Token\nImagine a city bus that refuses to let any new passengers board until every single person on the bus has reached their final destination, even if three people got off at the first stop and one person is riding all the way to the airport.\nThat is exactly how traditional\nstatic batching\noperates. If you batch four requests together that produce 50, 120, 240, and 1,024 tokens respectively, the GPU compute cores sit completely idle on three out of the four slots for hundreds of iterations, waiting for that single 1,024-token straggler to finally emit its\n<eos>\ntoken. These wasted cycles are known as\nGPU bubbles\n.\nvLLM implements\ncontinuous iteration-level batching\n(an architecture pioneered by Orca):\nThe scheduler makes decisions at the boundary of\nevery single forward pass\n, not at the boundary of whole requests.\nThe millisecond a sequence emits an end-of-sequence token, its physical memory blocks are freed back to the pool.\nA waiting request from the queue steps into that vacated slot on the very next token iteration. The GPU cores stay continuously saturated, and throughput jumps dramatically.\n4. Squeezing the Hardware: CUDA Graphs, Chunked Prefill, and FP8\nPagedAttention solves the memory footprint, but getting raw throughput out of modern NVIDIA Hopper silicon (H100/H200) requires tackling kernel dispatch overhead:\nThe Python Tax and CUDA Graphs (\nenforce_eager: false\n):\nIn standard PyTorch eager mode, generating a single token requires the Python runtime to launch dozens of individual GPU kernels in rapid succession across 60+ transformer layers (RMSNorm, QKV projection, RoPE, attention, SwiGLU, down-projection). On Hopper, a single-token GEMV kernel finishes in just 3 to 8 microseconds! But the CPU driver call to dispatch that kernel takes 10 to 15 microseconds. The GPU ends up spending more time waiting for Python to hand it work than actually doing the math. CUDA Graphs solve this by recording the entire sequence of operations into a static execution graph during warmup, allowing the GPU to replay the whole pipeline in a single dispatch.\nChunked Prefill:\nA massive 8,000-token prompt arriving during active generation used to cause a massive latency spike for everyone else. Chunked prefill breaks long prompts into manageable bites (e.g. 512 tokens) and interleaves them smoothly alongside decode tokens, keeping inter-token latency steady.\nFP8 Tensor Core GEMMs:\nRunning weights and activations in 8-bit floating point doubles effective memory bandwidth and unlocks Hopper's specialized Cutlass FP8 matrix cores.\nHot-Swapping LoRA Adapters:\nIn multi-task serving or RL training, you don't want to reboot your inference engine every time weights update. vLLM allows syncing LoRA adapter weights directly into the running worker processes over NCCL in milliseconds.\n5. Battle Scars from the Lab: Real Benchmarks from gft-studio\nSynthetic benchmark charts on Twitter are easy to fake. We wanted to see what happens when you push real engineering workloads through this stack. Below is empirical telemetry gathered on our research platform (\ngft-studio\n) running\nQwen3.6-27B\non dedicated NVIDIA H100 and H200 SXM clusters.\nBenchmark A: Hugging Face vs. vLLM on H100 (Controlled 1x H100 Smoke Test)\nIn Group Relative Policy Optimization (GRPO), models generate groups of rollouts against external environments. To cleanly isolate the engine, we ran a controlled 3-step test on an identical NVIDIA H100 80GB SXM GPU, keeping the prompt, base weights, random seed, and SQL task strictly identical:\nSampling Engine & Setup\nHardware\nSteps Completed\nAvg Gen Time / Step\nAvg Total Step Time\nObserved Speedup\nHugging Face Baseline\n1x H100 80GB\n3 / 3 (SUCCESS)\n283.7 s\n338.9 s\n1.00x (Baseline)\nvLLM + CUDA Graphs\n1x H100 80GB\n3 / 3 (SUCCESS)\n78.9 s\n143.5 s\n3.59x Gen Speedup (2.36x Step)\nOn identical silicon, vLLM cut generation wallclock from 283.7 seconds down to 78.9 seconds per step—a\n3.59x raw generation speedup\n. Total step time dropped by\n2.36x\n.\nCaveat:\nThis was a 3-step execution smoke test. While it cleanly isolates engine mechanics on identical hardware, it does not evaluate long-horizon policy convergence over 500 steps.\nBenchmark B: How Much Do CUDA Graphs Actually Matter on H200?\nTo measure host-side Python dispatch overhead in practice, we tested\nQwen3.6-27B\non NVIDIA H200 hardware with eager execution (\nenforce_eager: true\n) versus captured CUDA Graphs (\nenforce_eager: false\n):\nExperimental Arm\nExecution Mode\nenforce_eager Flag\nSteps & Status\nAvg Gen Time / Step\nObserved Ratio\nL0 Latent Think (Deterministic)\nPyTorch Eager Mode\nenforce_eager: true\n5 / 5 (SUCCESS)\n437.9 s / step\n1.00x (Baseline)\nL0 Latent Think (Deterministic)\nCUDA Graphs Replay\nenforce_eager: false\n5 / 5 (SUCCESS)\n101.3 s / step\n4.32x Speedup\nL2b Latent GRPO (Policy Gradients)\nPyTorch Eager Mode\nenforce_eager: true\n5 / 5 (SUCCESS)\n391.8 s / step\n1.00x (Baseline)\nL2b Latent GRPO (Policy Gradients)\nCUDA Graphs Replay\nenforce_eager: false\n50 / 50 (SUCCESS)\n132.7 s / step\n2.95x Speedup\nThe numbers speak for themselves: on single-token decode iterations, replaying pre-recorded CUDA graphs reduced generation time by\n2.95x to 4.32x\nsimply by removing host-side driver stalls.\nSupplemental Telemetry: Recorded Training Rows\nHistorical multi-hop routing runs (\nQwen3.6-27B\n, vLLM generation):\nRun ID\nLatent Variant\nEager Enforced?\nSteps\nGeneration / Step\nTotal / Step\n1788615027\nL0: answer-only objective\nYes\n5\n437.9 s\n582.7 s\n1788616147\nL0: answer-only objective\nNo\n5\n101.3 s\n247.9 s\n1788626786\nL2b: sampled latent objective\nYes\n5\n391.8 s\n567.9 s\n1788656686\nL2b: sampled latent objective\nNo\n50\n132.7 s\n291.6 s\nBenchmark C: Production Serving and Interconnect Bottlenecks\nIn an inference serving pilot using standard AIPerf workloads (564 input tokens, 128 output tokens), we pushed\nQwen3.8-27B\nunder increasing concurrency:\nHardware & Precision\nClient Concurrency\nTTFT (p50 / p99)\nOutput Throughput\nScaling Factor\n1x H100 BF16\n1\n100.7 ms / 106.8 ms\n48.4 tok/s\n1.0x\n1x H100 BF16\n4\n222.2 ms / 263.5 ms\n179.5 tok/s\n3.71x\n1x H100 BF16\n8\n406.7 ms / 487.1 ms\n307.4 tok/s\n6.35x\n2x H100 FP8 (Cutlass DP2)\n8\n172.9 ms / 274.5 ms\n483.2 tok/s\n9.98x\n2x H100 FP8 (Together TP2)\n8\n263.2 ms / 387.4 ms\n982.6 tok/s\n20.30x\nAn Engineering War Story:\nNotice the 982.6 tok/s peak under Tensor Parallelism (TP2). That speed was achieved on a single node connected via ultra-high-speed NVLink. Earlier in our testing, we attempted a custom cross-node TP2 setup over a standard VPC network interconnect. The result?\nThroughput collapsed to 75.8 tok/s\n! Unless your GPUs share high-bandwidth NVLink, do not run tensor parallelism across physical machines; network latency will decimate your throughput. Use data parallelism (independent workers) instead.\n6. Production Checklist: Running vLLM Without 3 AM Pages\nIf you are deploying vLLM in enterprise infrastructure, here is the architecture pattern we rely on:\nTwo-Tier Ingress Architecture:\nPlace a resilient gateway (like LiteLLM) in front to handle authentication, team quotas, and audit logging. Route traffic across backend vLLM workers using active queue-depth health checks.\nImmutable Local Storage:\nNever make your workers download 50 GB weight files from public object storage on startup. Pre-mount model checkpoints on local NVMe or high-speed read-only PVCs so pods boot in seconds.\nWatch Your Colocated Memory Budget:\nIf you run RL post-training where the trainer and the vLLM inference worker live on the same GPU, set\ngpu_memory_fraction: 0.35\n. This reserves ~28 GB for vLLM while leaving ~50 GB for PyTorch gradient activations and optimizer states. Neglecting this balance will trigger immediate CUDA OOM crashes the moment your training loss runs over a long trajectory.\nConclusion\nvLLM does not magically make models smarter, but it transforms LLM serving from a brittle, memory-starved script into a predictable, rock-solid engineering system. When you respect the hardware, the hardware delivers.\nInterested in specialized RL environments and post-training infrastructure? Explore our platform at\ng-ftech.com\nor launch a cluster on\ngft-studio\n.",
  "metadata": {
    "title": "High-Throughput LLM Inference & Training: A Deep Dive into vLLM - DEV Community",
    "description": "Editor's Note: Originally published on the g factor engineering blog. All benchmarks and telemetry... Tagged with ai, machinelearning, python, gpu.",
    "keywords": "ai, machinelearning, python, gpu, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/g_factor/high-throughput-llm-inference-training-a-deep-dive-into-vllm-5818",
    "og:title": "High-Throughput LLM Inference & Training: A Deep Dive into vLLM",
    "og:description": "Editor's Note: Originally published on the g factor engineering blog. All benchmarks and telemetry...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7jf8cruj0mucctnfqfdm.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "High-Throughput LLM Inference & Training: A Deep Dive into vLLM",
    "twitter:description": "Editor's Note: Originally published on the g factor engineering blog. All benchmarks and telemetry...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7jf8cruj0mucctnfqfdm.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq5693eu649x9foe39pwv.png",
      "bytes": 118344,
      "reused": true,
      "local_path": "attachments/81b67fe28cdb/images/003.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq5693eu649x9foe39pwv.png",
      "alt": "PagedAttention: reserve a contiguous strip vs. hand out physical pages",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fol45zm9q2rywuty83kvu.png",
      "bytes": 82530,
      "reused": true,
      "local_path": "attachments/81b67fe28cdb/images/004.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fol45zm9q2rywuty83kvu.png",
      "alt": "KV Cache Simulation: Reserve Maximum Allocation Policy",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fowdt9lp5z8dhjvbxreqn.png",
      "bytes": 80556,
      "reused": true,
      "local_path": "attachments/81b67fe28cdb/images/005.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fowdt9lp5z8dhjvbxreqn.png",
      "alt": "KV Cache Simulation: Allocate Blocks As Needed via PagedAttention",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.g-ftech.com/blog/vllm-throughput-deep-dive?utm_source=devto&utm_medium=syndication",
      "label": "g factor engineering blog",
      "type": "a"
    },
    {
      "href": "https://studio.g-ftech.com/",
      "label": "gft-studio",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq5693eu649x9foe39pwv.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fol45zm9q2rywuty83kvu.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fowdt9lp5z8dhjvbxreqn.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.g-ftech.com/?utm_source=devto&utm_medium=syndication",
      "label": "g-ftech.com",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:58Z",
  "source_url": "https://dev.to/g_factor/high-throughput-llm-inference-training-a-deep-dive-into-vllm-5818",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 14661,
  "expected_body_images": 3,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/81b67fe28cdb/pages/source.html",
  "local_html": "attachments/81b67fe28cdb/pages/article.html",
  "body_base_path": "attachments/81b67fe28cdb/pages/",
  "body_markdown_path": "attachments/81b67fe28cdb/pages/article.md"
}