{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "xbill",
  "title_listed": "Gemma 4 on a Tesla T4: QAT Weights Decode 1.79x Faster Than bf16",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/gde/gemma-4-on-a-tesla-t4-qat-weights-decode-179x-faster-than-bf16-2fi4",
  "publish_date": "2026-09-18",
  "corpus_stage": "",
  "notes": "",
  "id": "7803c957fd54",
  "collected_at": "2026-10-02T13:51:22",
  "title": "Gemma 4 on a Tesla T4: QAT Weights Decode 1.79x Faster Than bf16 - DEV Community",
  "body_markdown": "This article provides a step by step deployment guide for **Gemma 4 E2B\n* to a\nTesla T4\nhosted GPU enabled system. A suite of Python MCP tools is built to simplify management of the vLLM hosted deployment. The T4 is already attached to the Compute Engine VM the tools run on, so there is nothing to provision: the work is getting a Turing GPU to run Gemma 4, and then finding out which checkpoint it runs fastest.*\ngithub.com/xbill9/gemma4-dev/tree/main/gpu-vllm-t4-2b\nModels\ngoogle/gemma-4-E2B-it\n(bf16) and\ngoogle/gemma-4-E2B-it-qat-w4a16-ct\n(QAT, int4 weights)\nHardware\n1x Tesla T4, Turing, compute capability 7.5, 15360 MiB, 70 W\nHost\nCompute Engine\nn1-standard-2\n— 2 vCPU, 7.8 GB RAM,\nus-west2-b\nSoftware\nvLLM 0.29.0, torch 2.13.0+cu130\nResult\nQAT decodes at\n72.31 tok/s\nper stream against bf16's\n40.44\n, and serves\n215.91 tok/s\nat 8 streams against\n164.62\nWhere Do I Start?\nThe GPU is already there. A T4 attached to a Compute Engine VM needs no queued resource, no instance launch and no image, so this rig has no provisioning tools at all. Everything it ships is about the software on the box: which Python, which torch, which patch, and which checkpoint.\nThe MCP server exposes those steps as tools, in the order their failures get more expensive:\ncheck_host_capacity\n,\nverify_gpu_arch\n,\napply_turing_patch\n,\nverify_turing_patch\n,\nstart_vllm_server\n.\nAt This Point You Should Have…\nA Compute Engine VM with one Tesla T4 and the NVIDIA driver installed\nPython 3 for the MCP server, and a second interpreter for vLLM if your root disk is small\nRoom for the checkpoints on a disk that can hold them\nThe rig checked out from the repository above\nWhat the Machine Is\nmachine-type\n:\nn1-standard-2\nzone\n:\nus-west2-b\nnproc\n:\n2\nMemTotal\n:\n7614824 kB\nEnter fullscreen mode\nExit fullscreen mode\nTesla T4, 7.5, 15360 MiB, 615.71.09, 1590 MHz, 5001 MHz, 70.00 W, 70.00 W\nEnter fullscreen mode\nExit fullscreen mode\nTwo vCPU and 7.8 GB of RAM is a small host for this GPU, and the RAM shows up below when the weights load.\nThree Disks, Measured Separately\nMounted on    1B-blocks        Avail\n/           33570021376   8701743104\n/tmp         3898789888   3896131584\n/opt1      263086084096 200857096192\n~/.cache -> /opt1/cache\nEnter fullscreen mode\nExit fullscreen mode\nThe root disk is small,\n/tmp\nis its own 3.9 GB filesystem, and\n~/.cache\nis a symlink onto the large\n/opt1\nvolume. A single\ndf /\nreports the wrong answer for all three writes the install makes.\ncheck_host_capacity\nmeasures each target path on its own.\nTwo Pythons\npython3: Python 3.12.13 at /home/xbill_glitnir_com/.pyenv/shims/python3\nPYTHON_BIN: Python 3.13.5 at /usr/bin/python3.13\nEnter fullscreen mode\nExit fullscreen mode\nThe MCP server runs under\npython3\n. vLLM runs under\n/usr/bin/python3.13\nwith\nPYTHONUSERBASE=/opt1/pyuser\n, so its packages land on the big disk. Every tool that asks about torch or vLLM runs its check under that second interpreter.\nThe Install as Found\ntorch                                    2.11.0+cpu\nvllm                                     0.26.0\nEnter fullscreen mode\nExit fullscreen mode\nA CUDA build of vLLM sitting on the CPU build of torch.\nimport vllm\nfails on\nlibcudart.so.13\n, and a CPU torch reports an empty list for every GPU architecture. Two packages present, neither able to serve.\nWhich Torch\nThe fix is the CUDA 13 torch from the PyTorch index, installed with its dependencies. Installing it with\n--no-deps\nskips cuDNN and the NVIDIA runtime wheels, and\nimport torch\nthen fails:\nImportError: libcudnn.so.9: cannot open shared object file: No such file or directory\nEnter fullscreen mode\nExit fullscreen mode\nWith the dependencies in place,\nverify_gpu_arch\nasks the interpreter what it can run:\ntorch\n2.11\n.\n0\n+\ncu130\ntorch\n.\nversion\n.\ncuda\n13.0\ncuda\navailable\n:\nTrue\narch\nlist\n:\n[\n'\nsm_75\n'\n,\n'\nsm_80\n'\n,\n'\nsm_86\n'\n,\n'\nsm_90\n'\n,\n'\nsm_100\n'\n,\n'\nsm_120\n'\n]\ndevice\n:\nTesla\nT4\ncapability\n:\n(\n7\n,\n5\n)\nfp16\nmatmul\nok\n:\nTrue\nEnter fullscreen mode\nExit fullscreen mode\nsm_75\nis in the published wheel. Turing needs no source build.\nThen the Latest vLLM\nThe stack was upgraded to vLLM 0.29.0. vLLM pins torch to an exact version (\ntorch==2.13.0\n), so the upgrade moves torch with it:\ntorch                                    2.13.0+cu130\ntransformers                             5.17.0\ntriton                                   3.7.1\nvllm                                     0.29.0\nEnter fullscreen mode\nExit fullscreen mode\nThe 64 KiB Problem\nGemma 4 has two attention widths: 256 in its sliding-window layers and 512 in its global layers. vLLM forces its Triton attention backend for that mix, and at width 512 the kernel asks for more shared memory than Turing allows in one block.\napply_turing_patch\nclamps the tile sizes in the installed vLLM so they fit:\n✅ Patched, in the site-packages this host's `python3` imports.\n\n__FILE__/opt1/pyuser/lib/python3.13/site-packages/vllm/v1/attention/ops/triton_unified_attention.py\nCLAMP PRESENT\nOCCURRENCES 1\nEnter fullscreen mode\nExit fullscreen mode\nReinstalling vLLM removes the clamp. Run\nverify_turing_patch\nafter every vLLM change;\nstart_vllm_server\nrefuses to start until it reports the clamp present.\nLoading the Weights Needs Swap\nThe first start died during weight loading. The kernel log named the process:\nOut of memory: Killed process 14015 (VLLM::EngineCor) total-vm:27144376kB, anon-rss:3993204kB, file-rss:69212kB, shmem-rss:98540kB, UID:2110064466 pgtables:11488kB oom_score_adj:0\nEnter fullscreen mode\nExit fullscreen mode\nThe VM ships with no swap, and 7.8 GB of RAM is too little for vLLM to stage these weights. A 16 GB swapfile on the large disk fixes it:\n$\nsudo\nfallocate\n-l\n16G /opt1/swapfile\n&&\nsudo chmod\n600 /opt1/swapfile\n&&\nsudo\nmkswap /opt1/swapfile\n&&\nsudo\nswapon /opt1/swapfile\nSwap:          16383           0       16383\nEnter fullscreen mode\nExit fullscreen mode\nSwap use peaked at 6,761 MiB for bf16 and 4,770 MiB for QAT while the servers started. Add the swapfile to\n/etc/fstab\nso it survives a reboot.\nStarting the Server\n/usr/bin/python3.13\n-m\nvllm.entrypoints.openai.api_server\n--model\ngoogle/gemma-4-E2B-it-qat-w4a16-ct\n--host\n127.0.0.1\n--port\n8000\n--dtype\nfloat16\n--kv-cache-dtype\nauto\n--tensor-parallel-size\n1\n--gpu-memory-utilization\n0.9\n--max-model-len\n16384\n--max-num-seqs\n8\nEnter fullscreen mode\nExit fullscreen mode\n--dtype float16\nbecause Turing has no bfloat16 datapath. vLLM logs the cast and runs the bf16 weights as fp16.\nWhat the Engine Allocates\nUsing MarlinLinearKernel for CompressedTensorsWNA16\nModel loading took 8.02 GiB memory and 98.721582 seconds\nGPU KV cache size: 519,568 tokens, Maximum concurrency for 16,384 tokens per request: 31.71x\nEnter fullscreen mode\nExit fullscreen mode\nModel loading took 9.8 GiB memory and 157.183441 seconds\nGPU KV cache size: 315,974 tokens, Maximum concurrency for 16,384 tokens per request: 19.29x\nEnter fullscreen mode\nExit fullscreen mode\nThe first block is QAT, the second bf16. The QAT build packs its linear layers to 4 bits and keeps its embeddings and vision tower at full width, so its weights take 1.78 GiB less and the KV cache gets the difference.\nDoes It Serve?\n17*23 -> 391\nfacts -> The capital of Australia is Canberra and the chemical symbol for gold is Au.\nEnter fullscreen mode\nExit fullscreen mode\nSpot checks at temperature 0 on the QAT build. They show the 4-bit kernel producing correct text on this GPU; they are no substitute for an accuracy evaluation.\nWhy a Single Stream Tops Out Where It Does\nGenerating one token reads every weight the text path uses. The per-layer embedding table is a row lookup and the vision and audio towers sit idle, so what gets read is the decoder layers plus the tied embedding that serves as the output layer. From the safetensors headers:\n== models--google--gemma-4-E2B-it\n  bytes read per decode token: 4.597 GB\n  bound @ 277.0 GB/s measured stream:   60.3 tok/s\n== models--google--gemma-4-E2B-it-qat-w4a16-ct\n  bytes read per decode token: 1.862 GB\n  bound @ 277.0 GB/s measured stream:  148.8 tok/s\nEnter fullscreen mode\nExit fullscreen mode\n277.0 GB/s is the streaming read measured on this T4 part in an earlier run in the same repository. Dividing it by the bytes per token gives each build's ceiling. The QAT build reads 2.47x fewer bytes per token.\nThe Power Cap\n100 %, 1290 MHz, 70.99 W, 0x0000000000000004\n100 %, 1290 MHz, 69.97 W, 0x0000000000000004\n100 %, 1275 MHz, 70.05 W, 0x0000000000000004\nEnter fullscreen mode\nExit fullscreen mode\nDuring a single-stream bf16 decode the T4 sits at 100% utilization and at its 70 W limit, with the software power cap throttling the SM clock to 81% of its 1590 MHz maximum. The driver allows no higher limit on this card. The host CPU stays light: the engine uses 18-20% of one core.\nThe Sweep\nvllm bench serve\nwith random prompts, 128 output tokens, input lengths of 512 and 4096, concurrency 1, 4, 8 and 16, and three repeats per cell. Each cell and repeat gets its own seed, and both builds get the same seeds, so they see the same prompts. The server flags are identical for both. Sixteen cells, 48 runs, no failed requests, and the largest run-to-run variation in any cell is 2.0%.\n--max-num-seqs 8\ncaps the engine at eight running sequences, so at concurrency 16 the extra requests queue.\n512-Token Prompts\nc\nbf16 tok/s\nQAT tok/s\nbf16 per stream\nQAT per stream\n1\n37.04\n🥇 62.27\n40.44\n🥇 72.31\n4\n109.05\n🥇 157.49\n32.25\n🥇 54.50\n8\n164.62\n🥇 215.91\n25.49\n🥇 37.83\n16\n164.36\n🥇 213.79\n24.97\n🥇 34.69\nPer stream is 1000 divided by the median time per output token. QAT wins every cell:\n1.79x per stream at c=1\nand\n1.31x total at c=8\n. Both builds run on the same card at the same power cap. The QAT build moves fewer bytes per token.\nAgainst the bandwidth ceilings above, bf16 reaches 67% of its 60.3 tok/s and QAT 49% of its 148.8. QAT leaves more headroom, which fits a card that is also out of power.\n4096-Token Prompts\nc\nbf16 tok/s\nQAT tok/s\nbf16 TTFT\nQAT TTFT\n1\n12.11\n14.0\n7,199 ms\n7,243 ms\n4\n15.16\n15.82\n15,304 ms\n15,391 ms\n8\n15.66\n15.95\n16,779 ms\n16,879 ms\n16\n15.46\n15.75\n81,551 ms\n80,119 ms\nWith long prompts the two builds converge at about 15-16 tok/s. Time to first token sets the rate here, and it is the same for both: processing the prompt costs the same whichever way the weights are stored.\nPrefill Is the T4's Weak Point\nTime to first token at c=1 grows from 300 ms at 512 tokens to 7,199 ms at 4096 on bf16:\n24x the time for 8x the tokens\n. Growth that steep points at attention, whose cost rises with the square of the prompt. The likely suspect is the Turing tile clamp in the 512-wide global layers, which trades speed for fitting in 64 KiB. That is an inference; no profile has confirmed it yet.\nThe engine's prefix cache logged hits during the sweep, averaging about 8% even with a unique seed per cell. Both builds saw identical prompts, so the comparison between them holds, but the absolute prefill times may read slightly faster than a cold T4 delivers.\nMemory Never Binds\nAt c=8 with 4096-token prompts, eight sequences need 33,792 KV tokens. The bf16 pool holds 315,974, which is 9.4x that. At this serving shape GPU memory has room to spare with either build; QAT's 1.64x larger pool starts to matter only when the engine is allowed more concurrent sequences.\nAgainst an EC2 Twin\nThe same repository has an EC2\ng4dn\nrig on the same T4 part. It measured bf16 at 512-token prompts at 42.36 tok/s at c=1 and 242.47 tok/s at c=8. That run used vLLM 0.28.0 in a docker image, its own benchmark script with English filler prompts, and a 4 vCPU host. Its c=8 figure is 1.47x this run's bf16, and with four variables changed at once the cause of that gap is unknown.\nWhat Stops the Meter\n✅ Sent SIGTERM to vLLM (pid 18703). VRAM is released on exit.\nEnter fullscreen mode\nExit fullscreen mode\nstop_vllm_server\nfrees the GPU. The VM and its attached T4 bill by the hour whether vLLM runs or not; only stopping the VM stops the charge.\nSummary\nThe goal of this article was to serve Gemma 4 E2B on a Tesla T4 attached to a Compute Engine VM and find which checkpoint the card runs fastest. The key to the solution was measuring decode as a memory-bandwidth problem: the QAT w4a16 build reads 1.862 GB per token against bf16's 4.597, and vLLM 0.29's Marlin kernel runs it on Turing. The measured results were:\n72.31 tok/s\nper stream for QAT against\n40.44\nfor bf16 at 512-token prompts, 1.79x\n215.91 tok/s\ntotal at 8 streams for QAT against\n164.62\n, 1.31x\nAbout\n15-16 tok/s\nfor both builds at 4096-token prompts, where prompt processing sets the rate\n519,568\nKV tokens for QAT against\n315,974\nfor bf16; neither pool binds at eight sequences\nThe published cu130 torch wheel carries\nsm_75\n, and the 7.8 GB host needs swap to load the weights\nOne Tesla T4 on one\nn1-standard-2\nVM in\nus-west2-b\n, vLLM 0.29.0, three repeats per cell with the same seeds for both builds, largest variation in any cell 2.0%. The prefix cache logged hits despite unique seeds, which affects absolute prefill times but applies equally to both builds. The QAT build was spot-checked for correct output and was not evaluated for accuracy.\nThe strategy for using MCP for Tesla T4 deployment and benchmarking was validated with an incremental step by step approach.",
  "metadata": {
    "title": "Gemma 4 on a Tesla T4: QAT Weights Decode 1.79x Faster Than bf16 - DEV Community",
    "description": "A step by step deployment of Gemma 4 E2B with vLLM on a single Tesla T4 attached to a Compute Engine VM, and a measured comparison of the QAT w4a16 checkpoint against the bf16 reference on the same card. Tagged with gemma, vllm, cuda, machinelearning.",
    "keywords": "gemma, vllm, cuda, machinelearning, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/gde/gemma-4-on-a-tesla-t4-qat-weights-decode-179x-faster-than-bf16-2fi4",
    "og:title": "Gemma 4 on a Tesla T4: QAT Weights Decode 1.79x Faster Than bf16",
    "og:description": "A step by step deployment of Gemma 4 E2B with vLLM on a single Tesla T4 attached to a Compute Engine VM, and a measured comparison of the QAT w4a16 checkpoint against the bf16 reference on the same card.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fxbill9%2Fgemma4-dev%2Fmain%2Fgpu-vllm-t4-2b%2Fdevto-t4-qat-cover.74744fbf.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Gemma 4 on a Tesla T4: QAT Weights Decode 1.79x Faster Than bf16",
    "twitter:description": "A step by step deployment of Gemma 4 E2B with vLLM on a single Tesla T4 attached to a Compute Engine VM, and a measured comparison of the QAT w4a16 checkpoint against the bf16 reference on the same card.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fxbill9%2Fgemma4-dev%2Fmain%2Fgpu-vllm-t4-2b%2Fdevto-t4-qat-cover.74744fbf.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/7803c957fd54/code/body-000.txt",
      "chars": 82,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-001.txt",
      "chars": 74,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-002.txt",
      "chars": 172,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-003.txt",
      "chars": 121,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-004.txt",
      "chars": 100,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-005.txt",
      "chars": 86,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-006.txt",
      "chars": 189,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-007.txt",
      "chars": 197,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-008.txt",
      "chars": 192,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-009.txt",
      "chars": 180,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-010.txt",
      "chars": 177,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-011.txt",
      "chars": 261,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-012.txt",
      "chars": 202,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-013.txt",
      "chars": 150,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-014.txt",
      "chars": 99,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-015.txt",
      "chars": 263,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-016.txt",
      "chars": 135,
      "language": "text"
    },
    {
      "file": "attachments/7803c957fd54/code/body-017.txt",
      "chars": 62,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/xbill9/gemma4-dev/tree/main/gpu-vllm-t4-2b",
      "label": "github.com/xbill9/gemma4-dev/tree/main/gpu-vllm-t4-2b",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:29Z",
  "source_url": "https://dev.to/gde/gemma-4-on-a-tesla-t4-qat-weights-decode-179x-faster-than-bf16-2fi4",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 13198,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/7803c957fd54/pages/source.html",
  "local_html": "attachments/7803c957fd54/pages/article.html",
  "body_base_path": "attachments/7803c957fd54/pages/",
  "body_markdown_path": "attachments/7803c957fd54/pages/article.md"
}