{ "source_file": "0930", "source_platform": "Dev.to", "author": "xbill", "title_listed": "An MI300X Over MCP: What the Matrix Cores Execute, and What They Don't", "keywords": "", "domain": "大模型算法", "content_type": "工程实践", "is_paid_flag": "否", "url": "https://dev.to/gde/an-mi300x-over-mcp-what-the-matrix-cores-execute-and-what-they-dont-1me9", "publish_date": "2026-09-17", "corpus_stage": "", "notes": "", "id": "1ad4a20dbb18", "collected_at": "2026-10-02T13:51:25", "title": "An MI300X Over MCP: What the Matrix Cores Execute, and What They Don't - DEV Community", "body_markdown": "This article provides a step by step inventory and measurement of a single AMD Instinct MI300X on an AMD Developer Cloud hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of the droplet, and the same server is used to read the card's native numeric format support off the hardware.\nhttps://github.com/xbill9/amd-gputools\nThe workstation writing this has no AMD GPU and never will. The hardware is one MI300X droplet billed at $1.99 an hour, and everything that reaches it goes through twelve tag-scoped MCP tools for inventory, power, reboot, hardware scan, GPU state and remote execution.\nTwo of the readings those tools returned were wrong the first time, and both were corrected by parsing output instead of trusting a status. The format table at the end disagrees with AMD's published peaks in two places, and the disagreement is the result rather than a footnote to it.\nPrerequisites\nAn AMD Developer Cloud account with a GPU droplet already created.\ndevcloud.amd.com\nis\nDigitalOcean underneath — same v2 API, same droplet ids — and the token comes from the\nMy AMD Team\naccount, not a personal DigitalOcean one.\nThe droplet tagged. Every lookup in this server is scoped by\ntag_name\n, so an untagged\ndroplet is invisible to it and a tagged one belonging to someone else is not.\nDIGITALOCEAN_ACCESS_TOKEN\nin the environment, or in a mode 0600\n.env\n, or in\n~/ocean.txt\n. The same order is used by the server and by\nssh-droplet.sh\n.\nAn SSH key on the droplet as\nroot\n.\nBatchMode=yes\nthroughout, so a wrong key fails\nrather than waiting on a password prompt.\nPython 3 with\nhttpx\nand\npython-dotenv\nin the system interpreter. No virtualenv —\n.mcp.json\nlaunches the server with a bare\npython3\n.\nThe Box, In Detail\nEverything in these three tables was read off the machine by\nhardware_scan\n,\nrocminfo\nand\nlspci\nthrough the MCP server, on 2026-09-16.\nHost\nCPU\nINTEL(R) XEON(R) PLATINUM 8568Y+\nCores\n20 vCPU — 1 socket, 20 cores,\n1 thread per core\n(no SMT)\nCache\nL1d 640 KiB (20 instances), L2 80 MiB (20 instances)\nNUMA\n1 node\nRAM\n236 GB usable, 138 GB free, 90 GB in buff/cache\nDisk\n720 GB, 595 G free\nOS\nDebian 13 (trixie), kernel\n6.12.94+deb13-amd64\nDroplet\ngpu-mi300x1-192gb-devcloud\n, region\natl1\n, id\n601142018\nGPU\nDevice\nAMD Instinct MI300X\nVF\n(virtual function),\n[1002:74b5]\nat\n83:00.0\nTarget\ngfx942\n, ISA\namdgcn-amd-amdhsa--gfx942:sramecc+:xnack-\nCompute\n304 CUs\n, 4 SIMDs per CU, wavefront 64, max 32 waves per CU\nClock\n2100 MHz max\nWorkgroup\nmax 1024 threads, LDS (GROUP segment) 64 KB\nCacheline\n128 B\nLink\nPCIe\nGen5 x16\n—\nLnkSta: Speed 32GT/s, Width x16\nBAR\nRegion 0 is 256 G prefetchable, so the whole framebuffer is host-visible\nVBIOS\n113-M3000108-103\n, SKU\nM3000108\n, SMC firmware\n00.85.129.03\nUUID\nGPU-2896ede4ddd2a8b6\nGPU memory\n— the numbers that decide what fits:\nPool\nTotal\nUsed\nVRAM\n205,822,885,888 B =\n191.69 GiB\n180,717,051,904 B = 168.31 GiB (87%†)\nVIS_VRAM (host-visible)\n205,822,885,888 B = 191.69 GiB\n180,717,051,904 B = 168.31 GiB\nGTT (system memory aperture)\n126,676,250,624 B = 117.98 GiB\n21,327,872 B = 0.02 GiB\n† 87 is what\nrocm-smi\nreports; the division gives 87.8%. The tool truncates, and this article quotes the tool.\nTwo things worth reading off that table.\nVIS_VRAM equals VRAM\n: this is a large-BAR configuration, the entire 191.69 GiB is CPU-mappable, and no part of the framebuffer is hidden behind the old 256 MB window. And\nrocminfo\nreports three GLOBAL pools — coarse grained, fine grained and extended fine grained — each at 200,998,912 KB, which is the same 191.69 GiB described three ways, not three separate allocations.\nThe \"VF\" matters. This is a virtualized MI300X, not a bare card: firmware queries like ASD, PFP, MES and SOS answer\n\"Not supported on the given system\"\n, and\namd-smi\ncannot see partitioning. MEC (32948), RLC (65), SDMA (24), SMC and the RAS/XGMI TAs do report.\nThe Control Plane: Twelve Tools, Tag-Scoped\nget_help\nis the tool that describes the others, and its real output is the shape of the whole server:\n📡 amd-gputools — DigitalOcean control plane for this project's AMD MI300 droplets.\n\nScoped to droplets tagged `gemma`; SSH as `root`; checkout at `/opt/amd-gputools`.\n\n- list_droplets — List every droplet tagged for this project, with state and address.\n- droplet_status — Show one tagged droplet in detail, by numeric id or by name.\n- start_droplet — Power on a tagged droplet. No-op if it is already active.\n- stop_droplet — Power off a tagged droplet.\n- reboot_droplet — Reboot a tagged droplet.\n- action_status — Poll a droplet action returned by start_droplet or stop_droplet.\n- ssh_command — Print the ssh command for a tagged droplet, with its current address.\n- run_on_droplet — Run one shell command on a tagged droplet over SSH and return its output.\n- gpu_status — Report the AMD GPUs on a tagged droplet, and say why if there are none.\n- hardware_scan — Inventory a droplet: host, GPU, firmware, ROCm packages and installed tools.\n- list_gpu_sizes — List DigitalOcean GPU droplet sizes and their prices, cheapest first.\n- get_help — List the tools this server exposes.\n\nNo create or destroy tools, by design: both are dollar-per-hour decisions, so they stay a\ndeliberate step in the DigitalOcean console.\n\nPowering a droplet off does not stop DigitalOcean billing it.\nEnter fullscreen mode\nExit fullscreen mode\nThree design decisions are visible in that text.\nEvery lookup is scoped by tag.\nThe server resolves droplets through\nGET /v2/droplets?tag_name=gemma\n, so a tool cannot name, reboot or power off a droplet nobody tagged for it. The token in use is an account token with full reach; the tag is what keeps a reboot from landing on someone else's box.\nThere is no create and no destroy.\nNot an oversight. Creating an MI300X starts a meter and destroying one throws away state, and neither is a thing a model should be one tool call away from. The console is a fine place for a decision that costs dollars per hour.\nPowering off does not stop the billing.\nThis is the misconception the server exists to correct, so it is written into the tool output rather than a README. DigitalOcean reserves the resources of a stopped droplet and charges the full rate; only destroying it stops the meter.\nstop_droplet\nsays so in its return value every time, and a test enforces that the word survives refactoring.\nWorked Example 1: Inventory\nlist_droplets\ntakes no arguments. What comes back is the table, already tallied:\n| Droplet | ID | Size | Status | Public IP | Region |\n| --- | --- | --- | --- | --- | --- |\n|\n`debian-gpu-mi300x1-192gb-devcloud-atl1`\n|\n`601142018`\n| gpu-mi300x1-192gb-devcloud | active | 165.245.134.217 | atl1 |\n\n📡 1 droplet(s) tagged\n`gemma`\n: 1 active.\nEnter fullscreen mode\nExit fullscreen mode\nThe last line is the point.\nThe tool counts; the model quotes.\nOne droplet is trivially countable by eye, but the same code path handles twenty, and a model asked to count twenty rows is a coin flip. Every tool in this server that returns rows also returns the tally —\nlist_gpu_sizes\nsorts and names the cheapest,\nhardware_scan\ncounts present and missing tools,\ngpu_status\ncounts cards.\ndroplet_status 601142018\ndrills in, and ends with the sentence that costs money:\n**debian-gpu-mi300x1-192gb-devcloud-atl1** (`601142018`)\n\n- status: `active`\n- size: `gpu-mi300x1-192gb-devcloud` — 20 vCPU, 240 GB RAM, 720 GB disk\n- region: atl1\n- image: debian-13-x64\n- public IPv4: 165.245.134.217\n- tags: gemma\n- created: 2026-09-16T18:09:45Z\n\n✅ Reachable as `root@165.245.134.217`. Billing is running.\nEnter fullscreen mode\nExit fullscreen mode\nNote what is\nnot\nhere: a hardcoded IP anywhere in the server. Every tool resolves the address per call. A rebuilt droplet gets a new one, and a cached address turns into an SSH timeout that reads like a dead GPU.\nWorked Example 2: One Round Trip For The Whole Box\nhardware_scan\nanswers \"what can this box actually run\" in a single SSH round trip, instead of ten tool calls that each pay connection setup. It ships one shell script that prints\n<<>>\n-fenced sections, and the server parses them into a report. Abridged real output:\n## GPU\n\n- `/dev/kfd`: **present**\n- amdgpu module: loaded\n- pci: `83:00.0 Processing accelerators [1200]: ... [Instinct MI300X VF] [1002:74b5]`\n- gfx targets: `gfx942` (1 GPU agent(s))\n- device: AMD Instinct MI300X VF\n- vram: `card0,205822885888,180717051904`\n\n## Tools\n\n- present (9): `rocm-smi`, `rocminfo`, `amd-smi`, `clinfo`, `docker`, `python3`, `pip3`, `git`, `tmux`\n- **missing (2)**: `hipcc`, `podman`\n- ROCM-SMI version: 2.2.0+unknown\n- ROCM-SMI-LIB version: 7.2.0\n\n## ROCm packages (8)\n\nlibhsa-runtime64-1:amd64 6.1.2-3\nlibrocm-smi64-1 6.1.2-1\nrocm-smi 6.1.2-1\nrocminfo 6.1.2-2\n\n## Python\n\n- torch 2.9.1+rocm6.4 hip 6.4.43484-123eb5128 avail True\n- ModuleNotFoundError: No module named 'vllm'\nEnter fullscreen mode\nExit fullscreen mode\nThat last pair of lines is the whole vLLM-on-ROCm situation in two lines:\ntorch is installed and sees the GPU; vLLM is not installed and cannot be.\nThere is no prebuilt vLLM wheel for ROCm — PyPI's are CUDA builds, AMD's manylinux index carries torch and triton but no vllm, and building from source needs\nhipcc\nplus\nrocblas\n,\nhipblaslt\n,\nmiopen\nand\nrccl\n, of which Debian 13 ships exactly one (\nhipcc\n5.7.1, and the scan shows even that is absent here). vLLM comes from a container on this box, and the scan is what proves it rather than asserts it.\nOne parsing detail that cost a wrong number:\nrocminfo\nuses the key\nName:\nfor both the agent (\nName: gfx942\n) and its ISA (\nName: amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-\n). Matching on\ngfx\nalone counts one MI300X as\ntwo GPU agents\n. The scan only counts a bare\ngfx\ntarget.\nWorked Example 3: The Reboot That Is The Fix\nA freshly provisioned GPU droplet here does not work, and looks like a driver problem.\nlspci\nshows the card.\n/dev/dri/renderD128\nexists.\n/dev/kfd\ndoes not, because\namdgpu\nfailed to bind the VF and unloaded itself. Nothing needs installing — Debian's in-tree\namdgpu\nand the ROCm 6.1.2 userspace on the image are enough.\nreboot_droplet\n(\n\"\n601142018\n\"\n)\n📡\nRebooting\n`debian-gpu-mi300x1-192gb-devcloud-atl1`\n(\n`601142018`\n).\nAction\n``\nis\n`in-progress`\n.\nPoll\n`action_status`\n;\nSSH\ncame\nback\nabout\n20\ns\nafter\nthe\naction\ncompleted\nwhen\nthis\nwas\nmeasured\n.\nThen\ncheck\n`gpu_status`\n.\nEnter fullscreen mode\nExit fullscreen mode\nThe action id in that reply is what\naction_status\nthen polls, until the reboot action reports\ncompleted\nwith its start and finish timestamps. The sequence is three calls and no guessing: reboot, poll,\ngpu_status\n.\nAn hour went into diagnosing a driver stack that was fine, which is why that instruction now lives in the tool's own docstring where a model will read it before it starts debugging. These dmesg lines are benign on a VF and are\nnot\nthe problem:\nfailed to load amdgpu/psp_13_0_6_cap.bin (-2)\n,\nUnsupported TA type: 8\n,\nTMZ feature not supported\n.\nThe tool refuses to reboot anything that is not\nactive\n, pointing at\nstart_droplet\ninstead, and it is annotated\nDESTRUCTIVE\nin its MCP annotations — whatever is running on the card dies with the reboot.\nWorked Example 4: Two Readings That Were Wrong\nBoth were corrected the same way, and neither would have been caught by piping\nssh rocm-smi\ninto a model.\nThe exit code is not a signal.\nWith the driver uninitialised,\nrocm-smi\nprinted\nDriver not initialized (amdgpu not found in modules)\nto\nstderr\n, printed nothing to stdout, and\nexited 0\n.\namd-smi list\nprinted three ERROR lines and also exited 0. An early version of\ngpu_status\nbranched on the exit status and reported\n✅\nabove an empty table. Neither tool sets a useful exit code, so the current version does not consult the exit code at all: it parses\nrocm-smi --json\n, and if the parse does not yield at least one card it falls through to\namd-smi list\n, and if that fails too it says so and explains why.\nA key name taken from memory.\nOn 2026-09-16\ngpu_status\nreturned this, and it reads as an idle card with nothing running:\n| Card | Product | GPU use % | VRAM used % |\n| --- | --- | --- | --- |\n| card0 | Aqua Vanjaram [Instinct MI300X VF] | 0 | - |\n\n📡 1 GPU(s) reported by rocm-smi.\nEnter fullscreen mode\nExit fullscreen mode\nThe card was not idle. vLLM had been up for three hours holding\n168.31 GiB of the 191.69 GiB\n. The\n-\ncame from the summariser looking up\nGPU Memory Use (%)\n, a spelling that does not exist. ROCM-SMI 2.2.0 on this box emits:\n{\n\"card0\"\n:\n{\n\"Device Name\"\n:\n\"Aqua Vanjaram [Instinct MI300X VF]\"\n,\n\"GPU use (%)\"\n:\n\"0\"\n,\n\"GPU Memory Allocated (VRAM%)\"\n:\n\"87\"\n,\n\"Card Series\"\n:\n\"Aqua Vanjaram [Instinct MI300X VF]\"\n}}\nEnter fullscreen mode\nExit fullscreen mode\nGPU Memory Allocated (VRAM%)\n. The unit test that should have caught it used the invented key too, so it passed against a payload no machine produces. The fix is three words in a lookup list.\nA fixture written from memory tests the memory, not the tool\n, so the regression test now carries the verbatim payload above, and the fallback spellings stay for other rocm-smi builds.\nBoth have the same shape: a successful-looking result from a tool that had failed. A model handed raw\nrocm-smi\noutput reproduces both, because neither is visible without parsing what the tool printed. The control plane is where that parsing happens once.\nWorked Example 5: Driving The Workload Over The Same Server\nrun_on_droplet\nhands one command to the remote shell as a single argument — the local side never invokes a shell, and\nBatchMode=yes\nmeans a wrong key fails instead of hanging on a password prompt. It is how the workload gets checked:\nrun_on_droplet\n(\n\"601142018\"\n,\n\"docker ps --format '{{.Names}}\n\\t\n{{.Image}}\n\\t\n{{.Status}}'; curl -s localhost:8000/v1/models\"\n)\n✅ exited 0.\n\nvllm vllm/vllm-openai-rocm:nightly-rocm100 Up 3 hours\n{\n\"object\"\n:\n\"list\"\n,\n\"data\"\n:[\n{\n\"id\"\n:\n\"google/gemma-4-E2B-it\"\n,\n\"max_model_len\"\n:32768,...\n}]}\nEnter fullscreen mode\nExit fullscreen mode\nAnd\nlist_gpu_sizes(\"mi300\")\nreads the catalogue rather than trusting a slug, which surfaces an oddity:\nthe devcloud size this droplet runs is not in\nGET /v2/sizes\nat all.\nThe public catalogue lists\ngpu-mi300x1-192gb\nat $2.59/hr; the droplet's own\nsize_slug\nis\ngpu-mi300x1-192gb-devcloud\nat $1.99/hr, and it appears only on the droplet object. The tool shows the public catalogue and says which one it is showing.\nRules the Server Follows\nEach of these came out of a specific failure:\nReturn markdown with an emoji prefix\n—\n✅\nsuccess,\n❌\nerror,\n📡\nin progress,\n🛑\nstopping. Never a dict, never raw JSON for the model to interpret.\nErrors are returned, not raised.\nEvery tool ends in\nexcept Exception as exc: return _error(exc)\n. An exception escaping a tool kills the server process for every later call in the session. A test walks the module and fails if any tool lacks the handler.\nNever\nshell=True\n.\nEvery subprocess goes through one\nrun_command(cmd: list[str])\nusing\nasyncio.create_subprocess_exec\n. A test greps the source to keep it that way.\nPush the arithmetic into the tool.\nCounts, totals, sorts and the cheapest-size pick happen in Python. The model quotes a number it did not compute.\nPut the measurement in the docstring.\nreboot_droplet\nexplains the\n/dev/kfd\nsymptom;\ngpu_status\nexplains that the exit code is worthless and names the date it was measured. The docstring is the model's context window, and it is the cheapest place to prevent an hour of misdiagnosis.\nAnnotate write and destructive tools.\nREAD_ONLY\n,\nWRITE\n,\nDESTRUCTIVE\n— so a client can gate the ones that cost money or kill a running job.\nTests run offline.\nThe whole\nmcp\npackage is mocked before\nserver\nis imported, so no test needs a token or a network.\nWhy Not the Official DigitalOcean MCP Server?\nDigitalOcean ships one.\n@digitalocean/mcp\nis at 1.0.70 on npm, there is a hosted endpoint at\nhttps://droplets.mcp.digitalocean.com/mcp\nthat needs no local process at all, and the droplet area alone exposes 40 tools across 24 service areas. Seven of the twelve tools here have a direct equivalent in it.\nthis server\nofficial\nlist_droplets\ndroplet-list\ndroplet_status\ndroplet-get\nstart_droplet\npower-on-droplet\nstop_droplet\npower-off-droplet\nreboot_droplet\ndroplet-reboot\naction_status\ndroplet-action\nlist_gpu_sizes\nsize-list\nssh_command\n—\nrun_on_droplet\n—\ngpu_status\n—\nhardware_scan\n—\nThe four with no equivalent are the four that produced every measurement in this article.\nThe official server is a DigitalOcean API client, and the DigitalOcean API stops at the droplet object.\nSearching its repository for\nssh\n,\nexec\n,\nconsole\n,\ncommand\nor\nremote\nreturns nothing in the tree.\nWorked Example 3 is the clearest case. On a freshly provisioned droplet the v2 API reports\nstatus: active\nwhile the card is unusable, because\namdgpu\nfailed to bind the VF and unloaded, so\n/dev/kfd\ndoes not exist.\ndroplet-get\nreturns a healthy droplet in both states, because the difference is not in the droplet object.\nThe size catalogue lands the same way.\nsize-list\nreads\nGET /v2/sizes\n, which carries\ngpu-mi300x1-192gb\nat $2.59/hr and does not carry\ngpu-mi300x1-192gb-devcloud\nat all. Asked what this droplet costs, it has the wrong row available and the right one missing.\nOne difference is a design choice rather than a gap. The official server has\ndroplet-create\n,\ndroplet-delete\n, and tag-based bulk actions including\npower-off-droplets-tag\n, so a tag there is a selector for acting on many droplets at once. Here a tag is a boundary: every lookup is scoped by\ntag_name\n, and an untagged droplet is not addressable by any tool in the server. Both are right for what they are for. One manages a fleet; this one manages a single card that bills at $1.99 an hour with a model resident in 168 GiB of its memory.\nThe two compose rather than compete.\nThe account, the images, the volumes and the fleet belong to the official server, and there is no reason to reimplement any of it. What is worth writing yourself is the part past the API: remote execution, GPU state, and a hardware inventory that answers what the box can run rather than what it is billed as.\nWhat The Silicon Will Actually Compute\nThe control plane exists to report what the box is. The most consequential thing it reports is not capacity — it is\nwhich numeric formats the matrix cores execute natively\n, because that decides every quantization choice made afterwards, and it is the question a spec sheet answers least reliably.\nMeasured on the card 2026-09-16: 8192³ matmul, 30 iterations, torch 2.12.0+rocm10.0.0. The card was concurrently serving, so the absolute rates are depressed and\nthe ratios are the result\n.\ndtype\nms\nTFLOP/s\nSpec peak\n% of peak\nvs bf16\nbf16\n1.655\n664.3\n1307.4\n50.8%\n1.00x\nfp16\n1.660\n662.5\n1307.4\n50.7%\n1.00x\nfp8\ne4m3fnuz\n0.938\n1172.5\n2614.9\n44.8%\n1.77x\nint8\n2.413\n455.6\n2614.9\n17.4%\n0.69x\nFour readings, and three of them contradict something plausible.\nbf16 and fp16 are the same number because they are the same hardware.\nCDNA 3 runs both through one matrix pipeline at one peak of 1307.4 TFLOP/s. fp16 is not a cheaper precision you can trade down to here; it is the same speed with less exponent range. There is no decision to make between them.\nfp8 is the only format on this card faster than bf16\n— 1.77x measured against 2.00x theoretical, which is what a genuinely native path looks like once the parts of a matmul that are not the multiply are accounted for.\nint8 measured slower than bf16, and the spec says it should not have.\nAMD rates int8 at 2614.9 TOPS — exactly fp8's number, exactly double bf16's. On that basis the two are interchangeable and int8 is the better-supported choice. Measured, int8 reached 17.4% of its own peak and\n0.69x bf16\n. The silicon is not the problem; the kernels are, and\ntorch._int_mm\nis not reaching a tuned path on this stack.\nAn equal number in a spec table is not an equal number on the machine.\nfp4 is not present at all\n, and the capability list says otherwise.\nsupported_quantization\non this platform includes\nmxfp4\nand\nmxfp8\n, and torch 2.12 defines\ntorch.float4_e2m1fn_x2\n. Neither is a statement about this GPU. The hardware answers directly:\n>>>\ntorch\n.\n_scaled_mm\n(\na4\n,\nb4\n,\nscale_a\n=\ns\n,\nscale_b\n=\ns\n,\nout_dtype\n=\ntorch\n.\nbfloat16\n)\nNotImplementedError\n:\nBlock\n-\nwise\nscaling\nfor\nFloat8_e8m0fnu\nis\nonly\nsupported\non\ngfx950\n,\ngfx1250\nEnter fullscreen mode\nExit fullscreen mode\nvLLM gates on exactly the same boundary:\n@classmethod\ndef\nsupports_mx\n(\ncls\n)\n->\nbool\n:\nreturn\nany\n(\ngfx\nin\n_GCN_ARCH\nfor\ngfx\nin\n[\n\"\ngfx95\n\"\n,\n\"\ngfx1250\n\"\n])\nEnter fullscreen mode\nExit fullscreen mode\ngfx942\nis CDNA 3; MX formats arrive with CDNA 4 (\ngfx950\n, MI350X/MI355X). Here\nsupports_mx()\nis\nFalse\n, and rather than refusing, the stack falls back to an emulation kernel whose entire forward pass is:\ndq_w\n=\ndequant_mxfp4\n(\nlayer\n.\nweight\n,\nlayer\n.\nweight_scale\n,\nx\n.\ndtype\n)\nqdq_x\n=\nself\n.\nquant_dequant_func\n(\nx\n)\nreturn\nF\n.\nlinear\n(\nqdq_x\n,\ndq_w\n,\nbias\n)\nEnter fullscreen mode\nExit fullscreen mode\nIt widens the 4-bit weights back to bf16\non every forward pass\n, round-trips the activations through quantize-dequantize to reproduce fp4's error, and runs an ordinary bf16\nF.linear\n. The result is bf16 speed, no resident saving while computing, dequantization overhead on top, and the full quantization error. Its own log line calls it simulated. It answers whether a model survives fp4 before the hardware that runs fp4 is bought, and it is not a serving path.\nfnuz\nIs Not\nfn\nfp8 being native makes a ready-made fp8 checkpoint from the Hub look like the short path. It is not.\nCDNA 3 implements a different fp8 than Hopper and Blackwell do.\nFrom\ntorch.finfo\non this box:\ne4m3fn\n(NVIDIA, OCP)\ne4m3fnuz\n(CDNA 3)\nLargest finite value\n448.0\n240.0\nSmallest normal\n0.015625\n0.0078125\nThe exponent bias differs by one. The cleanest way to see what that means is to read a single byte as both types:\nbit pattern 0b01000000 as float8_e4m3fn -> 2.0\nbit pattern 0b01000000 as float8_e4m3fnuz -> 1.0\nEnter fullscreen mode\nExit fullscreen mode\nOne bit pattern, two values, a factor of two apart.\nReinterpreting an\ne4m3fn\ntensor as\ne4m3fnuz\nhalves every number in it. At the top of the range it is worse than halved:\nfloat8_e4m3fn round-trip [1.0, 2.0, 240.0, 448.0] -> [1.0, 2.0, 240.0, 448.0]\nfloat8_e4m3fnuz round-trip [1.0, 2.0, 240.0, 448.0] -> [1.0, 2.0, 240.0, nan]\nEnter fullscreen mode\nExit fullscreen mode\n448 is an ordinary weight on an H100 and is\nNaN\non an MI300X.\nAsking for the NVIDIA flavour fails at the call rather than silently:\nfloat8_e4m3fn: FAILED -> RuntimeError: HIPBLAS_STATUS_NOT_SUPPORTED\nfloat8_e4m3fnuz: _scaled_mm OK out=(4096, 4096) torch.bfloat16\nEnter fullscreen mode\nExit fullscreen mode\nvLLM agrees from its own side —\nis_fp8_fnuz()\nkeys on the string\n\"gfx94\"\n, and\nfp8_dtype()\nreturns\ntorch.float8_e4m3fnuz\n. So the safe route is to quantize\nonline, from the bf16 weights\n, and never to go looking for a checkpoint: the scales are then derived on the machine that will run them.\nGGUF Is Not Compiled In At All\nWorth stating because it fails differently from everything above — not gated, absent:\nCheck\nResult\n'gguf' in supported_quantization\nFalse\n'gguf' in QUANTIZATION_METHODS\n(the global registry)\nFalse\nvllm.model_executor.layers.quantization.gguf\nModuleNotFoundError\nggml/gguf symbols in\n_custom_ops\nnone\nOlder builds accepted\n--quantization gguf\n; this one has no such module. GGUF here means changing inference engine rather than passing a flag — and since its k-quants are weight-only anyway, dequantized in-kernel with the arithmetic still at bf16, it would not have unlocked a format the matrix cores lack.\nThe answer for this card is one line: fp8\ne4m3fnuz\n, quantized online, and nothing else.\nThree sources pointed elsewhere to get there — AMD's peak table rates int8 identically to fp8, the platform capability list includes\nmxfp4\n, and the Hub's fp8 checkpoints are the wrong flavour.\nThe Workload, Briefly\nThe card is not idle hardware. It serves\ngoogle/gemma-4-E2B-it\nthrough vLLM in a container, and that workload is what the memory figures below are measured against. The engine reports\ndtype=torch.bfloat16, quantization=None\n, matching the checkpoint's own\ndtype: bfloat16\n— so everything in this article is measured against an\nunquantized\nbaseline, and the 1.77x above is available and unclaimed.\nThe deployment itself is the subject of a companion article and is not repeated here. One finding from it carries across:\nthe newest ROCm vLLM image AMD publishes cannot load Gemma 4 at all.\nIt lacks\nGemma4ModelArchConfigConvertor\n, so a\nglobal\nread of\nhead_dim\nraises during config parsing, before the GPU is touched — Gemma 4 runs 256-wide heads on its sliding-attention layers and 512 on its full-attention ones, and transformers 5.15+ reports that as a per-layer attribute. The oldest image works because its transformers predates per-layer attributes; the nightly works because it has the convertor.\nThe broken build sits between the two that work\n, which is not a thing version intuition predicts.\nWhere The 191.69 GiB Goes\nThis is the number the control plane exists to report, so it is worth showing both sides of it. vLLM's own startup accounting for the serving run:\nFree memory on device (191.36/191.69 GiB) on startup. Desired GPU memory utilization is\n(0.9, 172.52 GiB). Actual usage is 11.88 GiB for consumed memory (weights + non-torch),\n5.59 GiB for peak activation, and 3.9 GiB for CUDAGraph memory. Current kv cache memory\nin use is 155.04 GiB.\n\nGPU KV cache size: 9,026,017 tokens\nMaximum concurrency for 32,768 tokens per request: 275.45x\ninit engine (profile, create kv cache, warmup model) took 84.91 s (compilation: 30.98 s)\nEnter fullscreen mode\nExit fullscreen mode\nClaim\nGiB\nWeights + non-torch\n11.88\nPeak activation\n5.59\nCUDA graphs\n3.90\nKV cache in use\n155.04\nvLLM's total\n176.41\nrocm-smi resident VRAM\n168.31 (87%)\nThe two disagree by about 8 GiB, and they are measured in different places — vLLM reports its own profiling arithmetic,\nrocm-smi\nreports what the driver sees resident. Both are quoted here rather than reconciled, because no experiment was run to explain the gap.\n9,026,017 tokens of KV works out to roughly 18 KB per token at bf16: E2B shares KV across 20 of its 35 layers and runs one KV head at 256, so 32k of context is cheap here.\n--max-model-len 32768\nis a workload choice, not a capacity one — the checkpoint supports 131072 and this card would hold it. vLLM also volunteers that\n--kv-cache-memory=182368940032\n(169.84 GiB) would fully use the card.\nText, thinking, tool calling and vision were each exercised against the live endpoint — riddle with\nreasoning\npopulated at 610 reasoning tokens, a\nget_weather\ncall returning\nfinish_reason: tool_calls\n, and a 158-byte synthetic red/blue checkerboard described correctly.\nAudio is unreachable on every image tried\n: E2B carries a conformer audio encoder, but none of these images ships the\nvllm[audio]\nextras (\nlibrosa\nand\nsoundfile\nare both missing), so\n--limit-mm-per-prompt '{\"audio\": 0}'\nis the setting that matches, and it also skips reserving encoder memory for a path that cannot be used.\nWhat This Does Not Cover\nNo serving throughput was measured. Nothing here is a serving benchmark: the KV figure is vLLM's own allocation report and no tokens per second number is claimed for the deployment.\nThe dtype table is a GEMM ratio rather than an end to end result. One matmul shape, on a card that was concurrently serving, which is why every absolute rate sits near half of peak and why the ratios are what is claimed. A 1.77x on\ntorch._scaled_mm\ndoes not imply 1.77x tokens per second, because attention and kernel launch overhead do not shrink with the weights and for a 2B model they are a large share of decode.\nThe int8 result is a statement about this software stack rather than about CDNA 3.\ntorch._int_mm\nreaching 17.4 percent of its own peak says a tuned kernel was not found, and a different library or a hand written MFMA path could plausibly close it. That was not attempted.\nThree further gaps: fp8 was measured but never deployed, so the serving arm ran bf16 throughout and no output quality evaluation of fp8 on this checkpoint exists; the 8 GiB accounting difference between vLLM and\nrocm-smi\nis reported from both sides and not explained; and the card is a virtual function, so 191.69 GiB is what SR-IOV exposes and partitioning on a bare card was not examined.\nSummary\nThe goal of this article was to manage one MI300X droplet entirely through MCP tools, and to read off the hardware which numeric formats its matrix cores execute. The key to the solution was parsing what each tool printed rather than branching on what it returned, and measuring every dtype on the card rather than reading a peak out of a table. The measured results were:\nfp8\ne4m3fnuz\nis the only format on this card faster than bf16 — 1172.5 against 664.3\nTFLOP/s, 1.77x, on an 8192³ matmul.\nbf16 and fp16 are one pipeline at one peak, 664.3 and 662.5 TFLOP/s, so there is no choice\nto make between them.\nint8 is rated 2614.9 TOPS, identical to fp8 and double bf16, and measured 455.6 TFLOP/s.\nThat is 17.4 percent of its own peak and 0.69x bf16.\nfp4 is not on\ngfx942\n.\nsupports_mx()\ngates it to\ngfx950\nand\ngfx1250\n, and the\nfallback kernel dequantizes to bf16 on every forward pass.\ne4m3fnuz\nis not\ne4m3fn\n. The same byte is 2.0 on an H100 and 1.0 here, 448 round-trips\nto\nNaN\n, and an fp8 checkpoint built for NVIDIA is not drop-in. Quantize online from bf16.\nrocm-smi\nand\namd-smi\nexit 0 when they fail. Parse the output; never branch on status.\nThe VRAM column read\n-\nfor hours because the summariser and its unit test both used a key\nname no machine emits. The real one is\nGPU Memory Allocated (VRAM%)\n.\nA freshly provisioned droplet needs one reboot.\n/dev/kfd\nis missing until then and\nnothing needs installing.\nPowering a droplet off does not stop the billing. Only destroying it does.\n191.69 GiB, 304 CUs and PCIe Gen5 x16 on one card, holding 9,026,017 tokens of KV at\n275.45x concurrency — all of it at bf16, with the 1.77x available and unclaimed.\nScope: one\ngpu-mi300x1-192gb-devcloud\ndroplet in\natl1\non 2026-09-16, a virtual function exposing all 304 CUs and 191.69 GiB. The dtype figures are a single 8192³ shape, 30 iterations after 5 warm-up, all four dtypes in one process against the same source buffers, measured inside the serving container on torch 2.12.0+rocm10.0.0 while vLLM held the card — which depresses every absolute rate and is why the ratios are quoted rather than the TFLOP/s. The bandwidth figure is bare metal torch 2.9.1+rocm6.4, a different stack, and is not comparable to the dtype rows. Spec peaks are AMD's published numbers and are not measurements. The $1.99 rate comes from AMD Developer Cloud and is not in\nGET /v2/sizes\n, which lists the public\ngpu-mi300x1-192gb\nat $2.59 instead. Every host, GPU and memory figure was read through the MCP server on the live box; the archived output is in\nevidence/\n.\nThe strategy for using MCP for single card platform management was validated with a incremental step by step approach.", "metadata": { "title": "An MI300X Over MCP: What the Matrix Cores Execute, and What They Don't - DEV Community", "description": "One AMD Instinct MI300X on AMD Developer Cloud, managed entirely through a tag-scoped Python MCP server, with every figure read off the card rather than a spec sheet. fp8 e4m3fnuz runs 1.77x bf16; int8, which AMD rates identically to fp8, runs 0.69x; fp4 is not on this silicon at all. One droplet, $1.99 an hour, and two readings that were wrong the first time. Tagged with amd, mcp, rocm, machinelearning.", "keywords": "amd, mcp, rocm, machinelearning, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/gde/an-mi300x-over-mcp-what-the-matrix-cores-execute-and-what-they-dont-1me9", "og:title": "An MI300X Over MCP: What the Matrix Cores Execute, and What They Don't", "og:description": "One AMD Instinct MI300X on AMD Developer Cloud, managed entirely through a tag-scoped Python MCP server, with every figure read off the card rather than a spec sheet. fp8 e4m3fnuz runs 1.77x bf16; int8, which AMD rates identically to fp8, runs 0.69x; fp4 is not on this silicon at all. One droplet, $1.99 an hour, and two readings that were wrong the first time.", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fxbill9%2Famd-gputools%2Fmain%2Fdevto-cover.403ae80c.jpg", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "An MI300X Over MCP: What the Matrix Cores Execute, and What They Don't", "twitter:description": "One AMD Instinct MI300X on AMD Developer Cloud, managed entirely through a tag-scoped Python MCP server, with every figure read off the card rather than a spec sheet. fp8 e4m3fnuz runs 1.77x bf16; int8, which AMD rates identically to fp8, runs 0.69x; fp4 is not on this silicon at all. One droplet, $1.99 an hour, and two readings that were wrong the first time.", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fxbill9%2Famd-gputools%2Fmain%2Fdevto-cover.403ae80c.jpg" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/1ad4a20dbb18/code/body-000.txt", "chars": 1311, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-001.txt", "chars": 256, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-002.txt", "chars": 324, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-003.txt", "chars": 750, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-004.txt", "chars": 252, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-005.txt", "chars": 160, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-006.txt", "chars": 187, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-007.txt", "chars": 279, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-008.txt", "chars": 172, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-009.txt", "chars": 108, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-010.txt", "chars": 135, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-011.txt", "chars": 98, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-012.txt", "chars": 158, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-013.txt", "chars": 134, "language": "text" }, { "file": "attachments/1ad4a20dbb18/code/body-014.txt", "chars": 471, "language": "text" } ], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://github.com/xbill9/amd-gputools", "label": "https://github.com/xbill9/amd-gputools", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:48:30Z", "source_url": "https://dev.to/gde/an-mi300x-over-mcp-what-the-matrix-cores-execute-and-what-they-dont-1me9", "papers": [], "actual_video_files": [], "body_chars": 31099, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/1ad4a20dbb18/pages/source.html", "local_html": "attachments/1ad4a20dbb18/pages/article.html", "body_base_path": "attachments/1ad4a20dbb18/pages/", "body_markdown_path": "attachments/1ad4a20dbb18/pages/article.md" }