{ "source_file": "0930", "source_platform": "Dev.to", "author": "xbill", "title_listed": "Serving Gemma 4 on an AMD MI300X: What $1.99 an Hour Buys", "keywords": "", "domain": "大模型算法", "content_type": "工程实践", "is_paid_flag": "否", "url": "https://dev.to/gde/serving-gemma-4-on-an-amd-mi300x-what-199-an-hour-buys-52h9", "publish_date": "2026-09-17", "corpus_stage": "", "notes": "", "id": "0d38a09dc0c2", "collected_at": "2026-10-02T13:51:24", "title": "Serving Gemma 4 on an AMD MI300X: What $1.99 an Hour Buys - DEV Community", "body_markdown": "This article provides a step by step deployment guide for **Gemma 4 E2B\n* to an\nAMD Instinct MI300X\nhosted GPU enabled system. A suite of Python MCP tools is built to simplify management of the vLLM hosted deployment. The card is reached through AMD Developer Cloud, which is DigitalOcean underneath, and the workstation driving it has no AMD GPU in it at all.*\ngithub.com/xbill9/gemma4-dev/tree/main/gpu-vllm-mi300x-2b\nModel\ngoogle/gemma-4-E2B-it\n, reference bf16 release\nHardware\n1x AMD Instinct MI300X,\ngfx942\n, CDNA 3, 191.7 GiB HBM3\nHost\ngpu-mi300x1-192gb-devcloud\n— 20 vCPU, 240 GB RAM, 720 GB disk, region\natl1\nControl plane\nAMD Developer Cloud / DigitalOcean v2 API — no\ngcloud\n, no EC2\nSoftware\nvllm/vllm-openai-rocm:nightly-rocm100\n, vLLM\n0.3.1.dev3+g0bfc7a15d\nRate\n$1.99 per hour\n, on-demand\nResult\n305 tok/s\nsingle stream,\n10,284 tok/s\nat 64 streams,\n2,713 tok/s per dollar-hour\nWhere Do I Start?\nThe card is a DigitalOcean GPU droplet reached through AMD Developer Cloud (\ndevcloud.amd.com\n) — same v2 API, same droplet ids, token from the My AMD Team account. Creating and destroying it are console actions, deliberately: both are dollar-per-hour decisions and neither belongs in a tool an agent can call.\nEverything after creation is scripted. The droplet carries a tag, every tool is scoped to that tag, and nothing in the toolkit can touch an instance that does not have it.\nAt This Point You Should Have…\nA GPU droplet running, tagged, and reachable by SSH key\nDIGITALOCEAN_ACCESS_TOKEN\nin a mode 0600\n.env\nbeside the server\nDocker on the droplet with\n/dev/kfd\nand\n/dev/dri\npresent\nA Hugging Face token on the droplet for the gated Gemma 4 weights\nPython 3 on your workstation, and no expectation of a GPU in it\nThere Is No GPU on the Machine You Run This From\nrocm-smi\n,\namd-smi\n,\nrocminfo\nand\nhipcc\ndo not exist locally and never will. A ROCm command in a local shell is a bug, not a check. Every reading in this article came back over SSH or through the DigitalOcean API.\nThat constraint shapes the whole toolkit. The MCP server runs on the workstation, holds no device, and reaches the hardware the same way you would by hand.\nThe MCP Server Speaks stdio\nOne file,\nserver.py\n, exposes 21 tools over stdio: droplet lifecycle, image checks, deploy, logs, benchmark cells, and the SRE probes below. Every subprocess goes through one\nrun_command(cmd: list[str])\nhelper using\nasyncio.create_subprocess_exec\n— never a shell.\n$\ngrep\n-c\n\"^@mcp.tool\"\nserver.py\n21\nEnter fullscreen mode\nExit fullscreen mode\nWhat the Droplet Is\n$\ndroplet_status\n📡 debian-gpu-mi300x1-192gb-devcloud-atl1 (601418522)\n\n- status: `active`\n- size: `gpu-mi300x1-192gb-devcloud` — 20 vCPU, 240 GB RAM, 720 GB disk\n- region: atl1\n- cost: $\n1.99/hr — billed\nwhile\npowered off, too\nEnter fullscreen mode\nExit fullscreen mode\nThe rate comes from the DigitalOcean v2 API's own\ndroplet.size.price_hourly\nfield, read at the time of the run, not from a pricing page.\nThe Card Before Anything Runs\n$ gpu_status\n✅ GPU reporting on\n`debian-gpu-mi300x1-192gb-devcloud-atl1`\n.\n\n| Card | Product | GPU use % | VRAM used % |\n| card0 | Aqua Vanjaram [Instinct MI300X VF] | 0 | 0 |\nEnter fullscreen mode\nExit fullscreen mode\nIt presents as an SR-IOV virtual function, which looks like a partition and is not one: all 304 compute units and the whole 191.7 GiB are there. Neither\nrocm-smi\nnor\namd-smi\nsets a useful exit code, so the tool parses the output and ignores the status code entirely.\nWhich Image\nvllm/vllm-openai-rocm:nightly-rocm100\n. Read the version out of the running container rather than trusting the tag, because a nightly tag moves under you:\n$\ndocker\nexec\nvllm python3\n-c\n\"import vllm;print(vllm.__version__)\"\n0.3.1.dev3+g0bfc7a15d\nEnter fullscreen mode\nExit fullscreen mode\nNot every ROCm vLLM image can load this checkpoint — Gemma 4 runs 256-wide heads on its sliding-attention layers and 512 on its full-attention ones, and an image whose vLLM predates that split cannot parse the config. The measured comparison of three images is in the\ncompanion article\n. Here the practical rule is enough: pin what you measured, and check a new image before you adopt it.\nStarting the Server\n$\ndeploy_vllm\n✅ Started `vllm` serving `google/gemma-4-E2B-it`.\n\n- image: `vllm/vllm-openai-rocm:nightly-rocm100`\n- context: 32768, gpu-memory-utilization 0.90\n- multimodal: `{\"image\": 4, \"audio\": 0}`\nEnter fullscreen mode\nExit fullscreen mode\nThe serve arguments, in full:\nvllm serve google/gemma-4-E2B-it\n--host\n0.0.0.0\n--port\n8000\n\\\n--max-model-len\n32768\n--gpu-memory-utilization\n0.90\n\\\n--enable-auto-tool-choice\n--reasoning-parser\ngemma4\n--tool-call-parser\ngemma4\n\\\n--chat-template\n/app/vllm/examples/tool_chat_template_gemma4.jinja\n\\\n--limit-mm-per-prompt\n'{\"image\": 4, \"audio\": 0}'\n--async-scheduling\nEnter fullscreen mode\nExit fullscreen mode\naudio\nis\n0\non purpose. E2B has a conformer audio encoder and no ROCm vLLM image ships the\nvllm[audio]\nextras, so a non-zero limit would allocate encoder memory for a path that cannot be used.\nUp Is Not Ready\ndocker run\nreturns in about a second. The endpoint does not answer for another two and a half minutes, while weights load,\ntorch.compile\nruns and graphs are captured.\n#\npolling /v1/models every 10s from the moment docker run returned\nREADY after 160s\n{\"object\":\"list\",\"data\":[{\"id\":\"google/gemma-4-E2B-it\",\"max_model_len\":32768,...}]}\nEnter fullscreen mode\nExit fullscreen mode\nThis is why\nserving_status\nreports the container and the endpoint as two separate facts. A running container is not a serving model, and conflating them turns a normal boot into a phantom hang.\n$\nserving_status\n✅ container: Up 3 minutes | endpoint: answering on `127.0.0.1:8000`\n served: `google/gemma-4-E2B-it`\nEnter fullscreen mode\nExit fullscreen mode\nDoes It Actually Serve?\nA flag being accepted is not evidence it did anything, so every modality is probed with a request whose correct answer is known in advance. The vision probe is a generated checkerboard rather than a fetched photo, so the expected answer is a fact about the image and not a caption.\n$\nverify_capabilities\n✅ 4/4 capabilities verified.\n\n| text | ✅ | The AMD MI300X is based on the CDNA 3 architecture. |\n| thinking | ✅ | 1404 chars, 447 reasoning tokens |\n| tool calling | ✅ | tool_calls → get_weather{\"city\": \"Reykjavik\"} |\n| vision | ✅ | alternating bright red and royal blue squares |\n\n📡 Audio is not probed — no ROCm vLLM image ships the `vllm[audio]` extras.\nEnter fullscreen mode\nExit fullscreen mode\nWhat the Engine Allocates\nThe boot log is worth reading before sizing anything, because the engine states its own arithmetic:\nAvailable KV cache memory: 155.04 GiB\nGPU KV cache size: 9,026,017 tokens\nMaximum concurrency for 32,768 tokens per request: 275.45x\nEnter fullscreen mode\nExit fullscreen mode\nNine million tokens of KV on one card. Hold that number — it decides which of the two ceilings below you actually hit.\nThe Sweep\nFour concurrencies against four context lengths, 128 output tokens throughout, three repeats per cell, load generated by vLLM's own bench client in a second container with no GPU device attached.\n$\npython3 benchmarking_suite.py\n--droplet\ndebian-gpu-mi300x1-192gb-devcloud-atl1\n\\\n--run-id 2026-09-17-vllm-sweep-mi300x-rerun --repeat 3 --seed-base 27000\n16 cells, 12 runnable, 4 infeasible at max_model_len 32768\nEnter fullscreen mode\nExit fullscreen mode\nThe 32,768 row is infeasible rather than missing: 32,768 input plus 128 output does not fit a 32,768 context, and a cell that cannot exist is recorded as such rather than dropped.\nOne detail is load-bearing.\nvllm bench serve\nderives its prompts from\n--seed\n, which defaults to 0, and this deployment caches prefixes — so two runs sharing a seed measure the cache, not the card. Every cell and every repeat here uses a seed no other run in the sweep uses.\nRaw Tokens per Second\nOutput tokens per second, median of three repeats, worst-cell coefficient of variation 7.96%:\ncontext\n1 stream\n4\n16\n64\n128\n340.9\n1,124.1\n3,569.1\n🥇\n10,284.0\n1,024\n305.1\n969.1\n2,653.6\n5,398.2\n8,192\n192.8\n445.5\n688.2\n702.7\nCounting prefill as well, the busiest cell moves\n48,583 tokens a second\n— 64 streams at 1,024 context. Single-stream latency at 1,024 context is 32.67 ms to first token and 3.04 ms per token after it.\nWhere the Ceiling Is\nTwo things happen as context grows, and only one of them is the one people plan for.\nTime per output token barely moves: 2.85 ms at 128 context, 3.04 at 1,024, 3.9 at 8,192. Time to first token rises steeply: 12.97 ms, 32.67 ms, 169.61 ms. Decode stays cheap; prefill gets expensive.\nSo the 8,192 row flattens — 688 tok/s at 16 streams, 703 at 64 — while the 128 row keeps climbing to 10,284.\nThe constraint at long context is prefill, not memory.\nThe heaviest cell in the grid wants 64 x 8,192 = 524,288 KV tokens against the 9,026,017 the engine allocated — 5.8% of the pool, by arithmetic.\nThat inverts the sizing rule the TPU rigs in this monorepo run on, where KV capacity is the thing you run out of first. On a 192 GB card serving a 2B model, it never becomes the binding constraint.\nAnd Price/Performance?\nThroughput divided by hourly rate, in=1024, out=128. Every row is the same checkpoint under vLLM, each from its own schema-valid report:\nrig\n$/hr\nbasis\n1\n4\n16\n64\nMI300X\n1.99\non-demand\n153\n487\n1,333\n🥇\n2,713\nTPU v5e-1\n0.5779\nspot\n🥇 208\n🥇 711\n🥇 1,572\n1,972\nTPU v6e-1\n2.97\non-demand\n67\n233\n587\n719\nNVIDIA L4\n0.94\nspot\n49\n187\n—\n—\nTokens per second per dollar-hour. The MI300X wins the busiest column outright and the v5e wins the other three — but those two rows are not priced the same way, which is the whole of the next section.\nThe Same Table, Priced the Same Way\nA spot rate against an on-demand rate is a discount, not a hardware result. Put all three on on-demand list and the ranking is uniform:\nrig\n$/hr\n1\n4\n16\n64\n🥇\nMI300X\n1.99\n153\n487\n1,333\n2,713\n🥈 TPU v5e-1\n1.20\n100\n342\n757\n950\n🥉 TPU v6e-1\n2.70\n74\n257\n646\n790\nAt matched pricing the MI300X returns\n1.4x to 2.9x\nthe tokens per dollar of a v5e — 1.53x, 1.42x, 1.76x and 2.86x across the four concurrencies, by arithmetic over the two measured rows above.\nThe Spot Asterisk\nThat advantage has one real qualifier: DigitalOcean publishes no preemptible tier for GPU droplets, so $1.99 is a floor. The v5e spot rate of $0.5779 per chip-hour is a purchasable option, and at 16 streams or fewer it buys more tokens per dollar than this card does.\nSo the honest form of the finding is conditional. If you want a non-preemptible box, or you keep dozens of streams busy, this is the best tokens-per-dollar measured across these rigs. If your traffic is light and you tolerate preemption, a spot TPU is cheaper.\nWhat Stops the Meter\nPowering the droplet off does\nnot\nstop DigitalOcean billing it. The resources stay reserved and the hourly rate keeps running; only destroying the droplet stops the meter.\nThat is why this toolkit ships no\ncreate\nand no\ndestroy\ntool. Both are dollar-per-hour decisions, they stay a deliberate step in the console, and a rate quoted per hour means nothing until you know which hours you are paying for.\n$\nstop_vllm\n✅ `docker rm -f vllm` exited 0.\nEnter fullscreen mode\nExit fullscreen mode\nThat frees the card's memory. It does not free your wallet.\nSummary\nThe goal of this article was to deploy Gemma 4 E2B to a single AMD Instinct MI300X and measure what its hourly rate returns in tokens. The key to the solution was a tag-scoped MCP toolkit that reaches the card over SSH and the DigitalOcean API, because the machine running it has no GPU. The measured results were:\n305 tok/s\nsingle stream at 1,024 context, 32.67 ms to first token\n10,284 tok/s\nat 64 streams and 128 context;\n48,583 tok/s\ncounting prefill\n2,713 tokens per second per dollar-hour\n, 1.4x to 2.9x a TPU v5e at matched on-demand pricing\n160 seconds\nfrom\ndocker run\nto an endpoint that answers\nKV pool of\n9,026,017 tokens\n, of which the heaviest cell used 5.8% — the long-context ceiling is prefill, not memory\nText, thinking, tool calling and vision all verified against the live endpoint; audio is unreachable on every ROCm image tried\nOne droplet, one MI300X, region\natl1\n, three repeats per cell with a unique prompt seed each, worst-cell coefficient of variation 7.96%. The comparison rows differ from this one in ways worth naming once: the TPU and L4 reports are single runs per cell on earlier vLLM builds, the v5e and L4 figures were measured on spot capacity while the MI300X and v6e were on-demand, and nothing else in this monorepo serves this checkpoint on AMD, so there is no twin to difference against.\nThe strategy for using MCP for AMD Instinct deployment and benchmarking was validated with an incremental step by step approach.", "metadata": { "title": "Serving Gemma 4 on an AMD MI300X: What $1.99 an Hour Buys - DEV Community", "description": "A step by step deployment of Gemma 4 E2B to a single AMD Instinct MI300X on AMD Developer Cloud, driven by Python MCP tools, and the throughput a 191.7 GiB card returns for its hourly rate. Tagged with amd, vllm, rocm, machinelearning.", "keywords": "amd, vllm, rocm, machinelearning, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/gde/serving-gemma-4-on-an-amd-mi300x-what-199-an-hour-buys-52h9", "og:title": "Serving Gemma 4 on an AMD MI300X: What $1.99 an Hour Buys", "og:description": "A step by step deployment of Gemma 4 E2B to a single AMD Instinct MI300X on AMD Developer Cloud, driven by Python MCP tools, and the throughput a 191.7 GiB card returns for its hourly rate.", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fxbill9%2Fgemma4-dev%2Fmain%2Fgpu-vllm-mi300x-2b%2Fdevto-dollar-hour-cover.02475528.jpg", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "Serving Gemma 4 on an AMD MI300X: What $1.99 an Hour Buys", "twitter:description": "A step by step deployment of Gemma 4 E2B to a single AMD Instinct MI300X on AMD Developer Cloud, driven by Python MCP tools, and the throughput a 191.7 GiB card returns for its hourly rate.", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fxbill9%2Fgemma4-dev%2Fmain%2Fgpu-vllm-mi300x-2b%2Fdevto-dollar-hour-cover.02475528.jpg" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/0d38a09dc0c2/code/body-000.txt", "chars": 36, "language": "text" }, { "file": "attachments/0d38a09dc0c2/code/body-001.txt", "chars": 226, "language": "text" }, { "file": "attachments/0d38a09dc0c2/code/body-002.txt", "chars": 175, "language": "text" }, { "file": "attachments/0d38a09dc0c2/code/body-003.txt", "chars": 90, "language": "text" }, { "file": "attachments/0d38a09dc0c2/code/body-004.txt", "chars": 201, "language": "text" }, { "file": "attachments/0d38a09dc0c2/code/body-005.txt", "chars": 341, "language": "text" }, { "file": "attachments/0d38a09dc0c2/code/body-006.txt", "chars": 168, "language": "text" }, { "file": "attachments/0d38a09dc0c2/code/body-007.txt", "chars": 120, "language": "text" }, { "file": "attachments/0d38a09dc0c2/code/body-008.txt", "chars": 397, "language": "text" }, { "file": "attachments/0d38a09dc0c2/code/body-009.txt", "chars": 133, "language": "text" }, { "file": "attachments/0d38a09dc0c2/code/body-010.txt", "chars": 219, "language": "text" }, { "file": "attachments/0d38a09dc0c2/code/body-011.txt", "chars": 44, "language": "text" } ], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://github.com/xbill9/gemma4-dev/tree/main/gpu-vllm-mi300x-2b", "label": "github.com/xbill9/gemma4-dev/tree/main/gpu-vllm-mi300x-2b", "type": "a" }, { "href": "https://github.com/xbill9/amd-gputools", "label": "companion article", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:48:29Z", "source_url": "https://dev.to/gde/serving-gemma-4-on-an-amd-mi300x-what-199-an-hour-buys-52h9", "papers": [], "actual_video_files": [], "body_chars": 12721, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/0d38a09dc0c2/pages/source.html", "local_html": "attachments/0d38a09dc0c2/pages/article.html", "body_base_path": "attachments/0d38a09dc0c2/pages/", "body_markdown_path": "attachments/0d38a09dc0c2/pages/article.md" }