{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Vuk Rosić",
  "title_listed": "DeepGEMM Essentials: High-Performance FP8 Matrix Multiplication",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/vuk_rosic/deepgemm-essentials-high-performance-fp8-matrix-multiplication-i02",
  "publish_date": "2025-07-07",
  "corpus_stage": "",
  "notes": "",
  "id": "183768e5260f",
  "collected_at": "2026-10-02T14:06:36",
  "title": "DeepGEMM Essentials: High-Performance FP8 Matrix Multiplication - DEV Community",
  "body_markdown": "DeepGEMM Essentials: High-Performance FP8 Matrix Multiplication\nGoogle Colab\nMaster these concepts and you'll be able to leverage cutting-edge FP8 acceleration on Hopper H1000, H200 & H800 GOUs!\nPart 1: Getting Started - Your First FP8 GEMM\nimport\ntorch\nimport\ndeep_gemm\n# Create simple input matrices\nm\n,\nn\n,\nk\n=\n128\n,\n256\n,\n512\nlhs\n=\ntorch\n.\nrandn\n((\nm\n,\nk\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\nrhs\n=\ntorch\n.\nrandn\n((\nn\n,\nk\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\noutput\n=\ntorch\n.\nempty\n((\nm\n,\nn\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\nprint\n(\nf\n\"\nLHS shape:\n{\nlhs\n.\nshape\n}\n\"\n)\n# [128, 512]\nprint\n(\nf\n\"\nRHS shape:\n{\nrhs\n.\nshape\n}\n\"\n)\n# [256, 512]\nprint\n(\nf\n\"\nOutput shape:\n{\noutput\n.\nshape\n}\n\"\n)\n# [128, 256]\nEnter fullscreen mode\nExit fullscreen mode\nWhat happened\n: We created the basic tensors for matrix multiplication: LHS × RHS^T = Output.\nPart 2: Understanding FP8 - Why It Matters\n# .numel() returns the total number of elements in a tensor\nsmall_tensor\n=\ntorch\n.\ntensor\n([[\n1\n,\n2\n,\n3\n],\n[\n4\n,\n5\n,\n6\n]])\nprint\n(\nf\n\"\nSmall tensor shape:\n{\nsmall_tensor\n.\nshape\n}\n\"\n)\nprint\n(\nf\n\"\nSmall tensor elements:\n{\nsmall_tensor\n.\nnumel\n()\n}\n\"\n)\n# 2 * 3 = 6\nmatrix_2d\n=\ntorch\n.\nrandn\n(\n50\n,\n20\n)\nprint\n(\nf\n\"\n2D matrix elements:\n{\nmatrix_2d\n.\nnumel\n()\n}\n\"\n)\n# 50 * 20 = 1000\n# Our actual tensors\nprint\n(\nf\n\"\nLHS elements:\n{\nlhs\n.\nnumel\n()\n}\n\"\n)\n# 128 * 512 = 65,536\nprint\n(\nf\n\"\nRHS elements:\n{\nrhs\n.\nnumel\n()\n}\n\"\n)\n# 256 * 512 = 131,072\nEnter fullscreen mode\nExit fullscreen mode\n# Regular BF16 GEMM (what you normally do)\nreference\n=\nlhs\n@\nrhs\n.\nt\n()\n# Standard PyTorch GEMM\n# Check memory usage\nbf16_memory\n=\nlhs\n.\nnumel\n()\n*\n2\n+\nrhs\n.\nnumel\n()\n*\n2\n# 2 bytes per BF16\nfp8_memory\n=\nlhs\n.\nnumel\n()\n*\n1\n+\nrhs\n.\nnumel\n()\n*\n1\n# 1 byte per FP8\nprint\n(\nf\n\"\nBF16 memory:\n{\nbf16_memory\n/\n1024\n**\n2\n:\n.\n1\nf\n}\nMB\n\"\n)\nprint\n(\nf\n\"\nFP8 memory:\n{\nfp8_memory\n/\n1024\n**\n2\n:\n.\n1\nf\n}\nMB\n\"\n)\nprint\n(\nf\n\"\nMemory saved:\n{\n(\n1\n-\nfp8_memory\n/\nbf16_memory\n)\n*\n100\n:\n.\n1\nf\n}\n%\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nKey insight\n: FP8 uses half the memory while maintaining good accuracy with proper scaling.\nPart 3: Converting to FP8 with Scaling\ndef\ncast_to_fp8_per_token\n(\nx\n:\ntorch\n.\nTensor\n):\n\"\"\"\nConvert tensor to FP8 with per-token (per-row) scaling\n\"\"\"\nassert\nx\n.\ndim\n()\n==\n2\nm\n,\nn\n=\nx\n.\nshape\n# Pad to 128-element boundaries (FP8 requirement)\npad_size\n=\n(\n128\n-\n(\nn\n%\n128\n))\n%\n128\nif\npad_size\n>\n0\n:\nx\n=\ntorch\n.\nnn\n.\nfunctional\n.\npad\n(\nx\n,\n(\n0\n,\npad_size\n),\nvalue\n=\n0\n)\n# Reshape for scaling calculation\nx_view\n=\nx\n.\nview\n(\nm\n,\n-\n1\n,\n128\n)\n# [m, n/128, 128]\n# Find max absolute value per 128-element block\nx_amax\n=\nx_view\n.\nabs\n().\nfloat\n().\namax\n(\ndim\n=\n2\n).\nclamp\n(\n1e-4\n)\n# [m, n/128]\n# Scale to FP8 range (448.0 is max representable value)\nfp8_data\n=\n(\nx_view\n*\n(\n448.0\n/\nx_amax\n.\nunsqueeze\n(\n2\n))).\nto\n(\ntorch\n.\nfloat8_e4m3fn\n)\nscale_factors\n=\n(\nx_amax\n/\n448.0\n)\nreturn\nfp8_data\n.\nview\n(\nm\n,\n-\n1\n)[:,\n:\nn\n],\nscale_factors\n# Convert our matrices\nlhs_fp8\n,\nlhs_scales\n=\ncast_to_fp8_per_token\n(\nlhs\n)\nprint\n(\nf\n\"\nOriginal:\n{\nlhs\n.\ndtype\n}\n, Converted:\n{\nlhs_fp8\n.\ndtype\n}\n\"\n)\nprint\n(\nf\n\"\nScale factors shape:\n{\nlhs_scales\n.\nshape\n}\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nCritical concept\n: Scaling prevents overflow and maintains precision in the limited FP8 range.\nPart 4: Block-wise Scaling for RHS\ndef\ncast_to_fp8_per_block\n(\nx\n:\ntorch\n.\nTensor\n):\n\"\"\"\nConvert tensor to FP8 with per-block scaling (128x128 blocks)\n\"\"\"\nm\n,\nn\n=\nx\n.\nshape\n# Pad to 128x128 blocks\npadded_m\n=\n((\nm\n+\n127\n)\n//\n128\n)\n*\n128\npadded_n\n=\n((\nn\n+\n127\n)\n//\n128\n)\n*\n128\nx_padded\n=\ntorch\n.\nzeros\n((\npadded_m\n,\npadded_n\n),\ndtype\n=\nx\n.\ndtype\n,\ndevice\n=\nx\n.\ndevice\n)\nx_padded\n[:\nm\n,\n:\nn\n]\n=\nx\n# Reshape into 128x128 blocks\nx_view\n=\nx_padded\n.\nview\n(\n-\n1\n,\n128\n,\nx_padded\n.\nsize\n(\n1\n)\n//\n128\n,\n128\n)\n# Find max per block\nx_amax\n=\nx_view\n.\nabs\n().\nfloat\n().\namax\n(\ndim\n=\n(\n1\n,\n3\n),\nkeepdim\n=\nTrue\n).\nclamp\n(\n1e-4\n)\n# Scale to FP8\nx_scaled\n=\n(\nx_view\n*\n(\n448.0\n/\nx_amax\n)).\nto\n(\ntorch\n.\nfloat8_e4m3fn\n)\nscale_factors\n=\n(\nx_amax\n/\n448.0\n).\nview\n(\nx_view\n.\nsize\n(\n0\n),\nx_view\n.\nsize\n(\n2\n))\nreturn\nx_scaled\n.\nview_as\n(\nx_padded\n)[:\nm\n,\n:\nn\n],\nscale_factors\n# Convert RHS with block scaling\nrhs_fp8\n,\nrhs_scales\n=\ncast_to_fp8_per_block\n(\nrhs\n)\nprint\n(\nf\n\"\nRHS FP8 shape:\n{\nrhs_fp8\n.\nshape\n}\n\"\n)\nprint\n(\nf\n\"\nRHS scales shape:\n{\nrhs_scales\n.\nshape\n}\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nWhy different scaling\n: LHS uses fine-grained scaling, RHS uses coarser blocks for efficiency.\nPart 5: Preparing Tensors for DeepGEMM\n# DeepGEMM requires specific tensor layouts\nfrom\ndeep_gemm\nimport\nget_col_major_tma_aligned_tensor\n# LHS scales must be transposed and TMA-aligned\nlhs_scales_aligned\n=\nget_col_major_tma_aligned_tensor\n(\nlhs_scales\n)\n# RHS scales must be contiguous\nassert\nrhs_scales\n.\nis_contiguous\n()\n# Package the inputs\nlhs_input\n=\n(\nlhs_fp8\n,\nlhs_scales_aligned\n)\nrhs_input\n=\n(\nrhs_fp8\n,\nrhs_scales\n)\nprint\n(\n\"\n✓ Tensors prepared for DeepGEMM\n\"\n)\nprint\n(\nf\n\"\nLHS scales alignment:\n{\nlhs_scales_aligned\n.\nstride\n()\n}\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nTMA requirement\n: Tensor Memory Accelerator needs specific memory alignment for optimal performance.\nPart 6: Your First DeepGEMM Call\n# Perform the FP8 GEMM\ndeep_gemm\n.\ngemm_fp8_fp8_bf16_nt\n(\nlhs_input\n,\nrhs_input\n,\noutput\n)\n# Verify correctness\nreference\n=\nlhs\n@\nrhs\n.\nt\n()\nerror\n=\ntorch\n.\nabs\n(\noutput\n-\nreference\n).\nmax\n().\nitem\n()\nrelative_error\n=\n(\nerror\n/\ntorch\n.\nabs\n(\nreference\n).\nmax\n().\nitem\n())\n*\n100\nprint\n(\nf\n\"\nMax absolute error:\n{\nerror\n:\n.\n6\nf\n}\n\"\n)\nprint\n(\nf\n\"\nRelative error:\n{\nrelative_error\n:\n.\n3\nf\n}\n%\n\"\n)\nprint\n(\n\"\n✓ FP8 GEMM completed successfully!\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nResult\n: High-performance FP8 matrix multiplication with automatic kernel optimization.\nPart 7: Understanding the Performance Gain\nimport\ntime\ndef\nbenchmark_gemm\n(\nfunc\n,\n*\nargs\n,\nnum_runs\n=\n10\n):\n# Warmup\nfor\n_\nin\nrange\n(\n3\n):\nfunc\n(\n*\nargs\n)\ntorch\n.\ncuda\n.\nsynchronize\n()\n# Timing\nstart\n=\ntime\n.\ntime\n()\nfor\n_\nin\nrange\n(\nnum_runs\n):\nfunc\n(\n*\nargs\n)\ntorch\n.\ncuda\n.\nsynchronize\n()\nreturn\n(\ntime\n.\ntime\n()\n-\nstart\n)\n/\nnum_runs\n# Benchmark both versions\nfp8_time\n=\nbenchmark_gemm\n(\ndeep_gemm\n.\ngemm_fp8_fp8_bf16_nt\n,\nlhs_input\n,\nrhs_input\n,\noutput\n)\nbf16_time\n=\nbenchmark_gemm\n(\nlambda\nx\n,\ny\n,\nout\n:\nout\n.\ncopy_\n(\nx\n@\ny\n.\nt\n()),\nlhs\n,\nrhs\n,\nreference\n)\n# Calculate throughput (TFLOPS)\nops\n=\n2\n*\nm\n*\nn\n*\nk\n# Multiply-accumulate operations\nfp8_tflops\n=\nops\n/\nfp8_time\n/\n1e12\nbf16_tflops\n=\nops\n/\nbf16_time\n/\n1e12\nprint\n(\nf\n\"\nFP8 GEMM:\n{\nfp8_time\n*\n1000\n:\n.\n2\nf\n}\nms (\n{\nfp8_tflops\n:\n.\n1\nf\n}\nTFLOPS)\n\"\n)\nprint\n(\nf\n\"\nBF16 GEMM:\n{\nbf16_time\n*\n1000\n:\n.\n2\nf\n}\nms (\n{\nbf16_tflops\n:\n.\n1\nf\n}\nTFLOPS)\n\"\n)\nprint\n(\nf\n\"\nSpeedup:\n{\nbf16_time\n/\nfp8_time\n:\n.\n1\nf\n}\nx\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nPerformance\n: FP8 can achieve 2-3x speedup on modern GPUs while using half the memory.\nPart 8: Grouped GEMM - Processing Multiple Experts\n# Simulate MoE (Mixture of Experts) scenario\nnum_experts\n=\n4\ntokens_per_expert\n=\n[\n128\n,\n96\n,\n112\n,\n144\n]\n# Variable tokens per expert\nexpert_dim\n=\n512\n# Create contiguous tensor for all tokens\ntotal_tokens\n=\nsum\n(\ntokens_per_expert\n)\nalignment\n=\ndeep_gemm\n.\nget_m_alignment_for_contiguous_layout\n()\n# 128\n# Align each expert's token count\naligned_tokens\n=\n[((\nt\n+\nalignment\n-\n1\n)\n//\nalignment\n)\n*\nalignment\nfor\nt\nin\ntokens_per_expert\n]\ntotal_aligned\n=\nsum\n(\naligned_tokens\n)\nprint\n(\nf\n\"\nOriginal tokens:\n{\ntokens_per_expert\n}\n\"\n)\nprint\n(\nf\n\"\nAligned tokens:\n{\naligned_tokens\n}\n\"\n)\nprint\n(\nf\n\"\nTotal aligned:\n{\ntotal_aligned\n}\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nMoE insight\n: Different experts process different numbers of tokens - grouping improves efficiency.\nPart 9: Setting Up Grouped GEMM Data\n# Create inputs for grouped GEMM\nlhs_grouped\n=\ntorch\n.\nrandn\n((\ntotal_aligned\n,\nk\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\nrhs_grouped\n=\ntorch\n.\nrandn\n((\nnum_experts\n,\nn\n,\nk\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\noutput_grouped\n=\ntorch\n.\nempty\n((\ntotal_aligned\n,\nn\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\n# Create mapping tensor\nm_indices\n=\ntorch\n.\nempty\n(\ntotal_aligned\n,\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nint32\n)\nstart\n=\n0\nfor\nexpert_id\n,\n(\norig_tokens\n,\naligned_tokens\n)\nin\nenumerate\n(\nzip\n(\ntokens_per_expert\n,\naligned_tokens\n)):\n# Real tokens get expert ID\nm_indices\n[\nstart\n:\nstart\n+\norig_tokens\n]\n=\nexpert_id\n# Padding tokens get -1 (ignored)\nm_indices\n[\nstart\n+\norig_tokens\n:\nstart\n+\naligned_tokens\n]\n=\n-\n1\nstart\n+=\naligned_tokens\nprint\n(\nf\n\"\nMapping tensor shape:\n{\nm_indices\n.\nshape\n}\n\"\n)\nprint\n(\nf\n\"\nExpert assignments:\n{\nm_indices\n[\n:\n20\n]\n}\n\"\n)\n# First 20 tokens\nEnter fullscreen mode\nExit fullscreen mode\nMapping\n: Each token knows which expert should process it.\nPart 10: Converting Grouped Data to FP8\n# Convert LHS (same as before)\nlhs_grouped_fp8\n,\nlhs_grouped_scales\n=\ncast_to_fp8_per_token\n(\nlhs_grouped\n)\nlhs_grouped_scales\n=\nget_col_major_tma_aligned_tensor\n(\nlhs_grouped_scales\n)\n# Convert each expert's RHS separately\nrhs_grouped_fp8\n=\ntorch\n.\nempty_like\n(\nrhs_grouped\n,\ndtype\n=\ntorch\n.\nfloat8_e4m3fn\n)\nrhs_grouped_scales\n=\ntorch\n.\nempty\n((\nnum_experts\n,\n(\nn\n+\n127\n)\n//\n128\n,\n(\nk\n+\n127\n)\n//\n128\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nfloat32\n)\nfor\nexpert_id\nin\nrange\n(\nnum_experts\n):\nrhs_grouped_fp8\n[\nexpert_id\n],\nrhs_grouped_scales\n[\nexpert_id\n]\n=\ncast_to_fp8_per_block\n(\nrhs_grouped\n[\nexpert_id\n])\n# Package inputs\nlhs_grouped_input\n=\n(\nlhs_grouped_fp8\n,\nlhs_grouped_scales\n)\nrhs_grouped_input\n=\n(\nrhs_grouped_fp8\n,\nrhs_grouped_scales\n)\nprint\n(\n\"\n✓ Grouped data converted to FP8\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nExpert-wise\n: Each expert has its own scaling factors for optimal precision.\nPart 11: Running Grouped GEMM\n# Perform grouped GEMM\ndeep_gemm\n.\nm_grouped_gemm_fp8_fp8_bf16_nt_contiguous\n(\nlhs_grouped_input\n,\nrhs_grouped_input\n,\noutput_grouped\n,\nm_indices\n)\n# Verify by computing reference\nreference_grouped\n=\ntorch\n.\nzeros_like\n(\noutput_grouped\n)\nstart\n=\n0\nfor\nexpert_id\n,\naligned_tokens\nin\nenumerate\n(\naligned_tokens\n):\nend\n=\nstart\n+\naligned_tokens\nreference_grouped\n[\nstart\n:\nend\n]\n=\nlhs_grouped\n[\nstart\n:\nend\n]\n@\nrhs_grouped\n[\nexpert_id\n].\nt\n()\nstart\n=\nend\n# Mask out padding tokens for comparison\nvalid_mask\n=\n(\nm_indices\n!=\n-\n1\n).\nunsqueeze\n(\n1\n)\noutput_masked\n=\ntorch\n.\nwhere\n(\nvalid_mask\n,\noutput_grouped\n,\ntorch\n.\nzeros_like\n(\noutput_grouped\n))\nreference_masked\n=\ntorch\n.\nwhere\n(\nvalid_mask\n,\nreference_grouped\n,\ntorch\n.\nzeros_like\n(\nreference_grouped\n))\nerror\n=\ntorch\n.\nabs\n(\noutput_masked\n-\nreference_masked\n).\nmax\n().\nitem\n()\nprint\n(\nf\n\"\nGrouped GEMM error:\n{\nerror\n:\n.\n6\nf\n}\n\"\n)\nprint\n(\n\"\n✓ Grouped GEMM completed successfully!\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nValidation\n: Compare against standard computation to ensure correctness.\nPart 12: Weight Gradient GEMM\n# For training: compute weight gradients\ndef\nsetup_weight_gradient\n():\nm_grad\n,\nk_grad\n,\nn_grad\n=\n256\n,\n1024\n,\n512\n# Activations (forward pass)\nactivations\n=\ntorch\n.\nrandn\n((\nm_grad\n,\nk_grad\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\n# Gradient w.r.t. output (from backprop)\ngrad_output\n=\ntorch\n.\nrandn\n((\nm_grad\n,\nn_grad\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\n# Weight gradient accumulator (typically has residual)\nweight_grad\n=\ntorch\n.\nrandn\n((\nn_grad\n,\nk_grad\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nfloat\n)\n*\n0.1\nreturn\nactivations\n,\ngrad_output\n,\nweight_grad\nactivations\n,\ngrad_output\n,\nweight_grad\n=\nsetup_weight_gradient\n()\n# Convert to FP8\nact_fp8\n,\nact_scales\n=\ncast_to_fp8_per_token\n(\nactivations\n)\ngrad_fp8\n,\ngrad_scales\n=\ncast_to_fp8_per_token\n(\ngrad_output\n)\n# Prepare inputs (both need transposed scales)\nact_input\n=\n(\nact_fp8\n,\nget_col_major_tma_aligned_tensor\n(\nact_scales\n))\ngrad_input\n=\n(\ngrad_fp8\n,\nget_col_major_tma_aligned_tensor\n(\ngrad_scales\n))\nprint\n(\nf\n\"\nWeight gradient shape:\n{\nweight_grad\n.\nshape\n}\n\"\n)\nprint\n(\nf\n\"\nAccumulator dtype:\n{\nweight_grad\n.\ndtype\n}\n\"\n)\n# FP32 for precision\nEnter fullscreen mode\nExit fullscreen mode\nTraining context\n: Weight gradients accumulate many small updates - need FP32 precision.\nPart 13: Computing Weight Gradients\n# Compute weight gradients with accumulation\noriginal_grad\n=\nweight_grad\n.\nclone\n()\ndeep_gemm\n.\nwgrad_gemm_fp8_fp8_fp32_nt\n(\ngrad_input\n,\nact_input\n,\nweight_grad\n)\n# Verify: grad_output^T @ activations + original_grad\nreference_update\n=\ngrad_output\n.\nfloat\n().\nt\n()\n@\nactivations\n.\nfloat\n()\nexpected_grad\n=\noriginal_grad\n+\nreference_update\nerror\n=\ntorch\n.\nabs\n(\nweight_grad\n-\nexpected_grad\n).\nmax\n().\nitem\n()\nrelative_error\n=\nerror\n/\ntorch\n.\nabs\n(\nexpected_grad\n).\nmax\n().\nitem\n()\nprint\n(\nf\n\"\nWeight gradient error:\n{\nerror\n:\n.\n6\nf\n}\n\"\n)\nprint\n(\nf\n\"\nRelative error:\n{\nrelative_error\n*\n100\n:\n.\n3\nf\n}\n%\n\"\n)\nprint\n(\n\"\n✓ Weight gradient computation successful!\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nAccumulation\n: New gradients are added to existing values, enabling mini-batch training.\nPart 14: Performance Monitoring\ndef\nanalyze_gemm_performance\n(\nm\n,\nn\n,\nk\n,\noperation\n=\n\"\nforward\n\"\n):\n# Theoretical peak performance\n# H100 has ~1600 TFLOPS FP8 peak\nops\n=\n2\n*\nm\n*\nn\n*\nk\npeak_time\n=\nops\n/\n1600e12\n# Theoretical minimum time\n# Memory bandwidth\nfp8_bytes\n=\n(\nm\n*\nk\n+\nn\n*\nk\n)\n*\n1\n# FP8 inputs\nbf16_bytes\n=\nm\n*\nn\n*\n2\n# BF16 output\nscale_bytes\n=\n((\nm\n*\nk\n)\n//\n128\n+\n(\nn\n*\nk\n)\n//\n128\n)\n*\n4\n# FP32 scales\ntotal_bytes\n=\nfp8_bytes\n+\nbf16_bytes\n+\nscale_bytes\n# H100 has ~3TB/s memory bandwidth\nbandwidth_time\n=\ntotal_bytes\n/\n3e12\nprint\n(\nf\n\"\n\\n\n{\noperation\n.\nupper\n()\n}\nGEMM Analysis (M=\n{\nm\n}\n, N=\n{\nn\n}\n, K=\n{\nk\n}\n):\n\"\n)\nprint\n(\nf\n\"\nOperations:\n{\nops\n/\n1e9\n:\n.\n1\nf\n}\nGigaOps\n\"\n)\nprint\n(\nf\n\"\nCompute bound time:\n{\npeak_time\n*\n1000\n:\n.\n2\nf\n}\nms\n\"\n)\nprint\n(\nf\n\"\nMemory bound time:\n{\nbandwidth_time\n*\n1000\n:\n.\n2\nf\n}\nms\n\"\n)\nprint\n(\nf\n\"\nBottleneck:\n{\n'\nCompute\n'\nif\npeak_time\n>\nbandwidth_time\nelse\n'\nMemory\n'\n}\n\"\n)\n# Analyze our configurations\nanalyze_gemm_performance\n(\n128\n,\n256\n,\n512\n,\n\"\nforward\n\"\n)\nanalyze_gemm_performance\n(\n256\n,\n512\n,\n1024\n,\n\"\nweight_grad\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nPerformance tuning\n: Understanding compute vs memory bottlenecks helps optimize configurations.\nPart 15: Advanced Configuration\n# Control SM utilization for better efficiency\noriginal_sms\n=\ndeep_gemm\n.\nget_num_sms\n()\nprint\n(\nf\n\"\nDefault SMs:\n{\noriginal_sms\n}\n\"\n)\n# Use fewer SMs for smaller problems to save power\ndeep_gemm\n.\nset_num_sms\n(\noriginal_sms\n//\n2\n)\nprint\n(\nf\n\"\nReduced SMs:\n{\ndeep_gemm\n.\nget_num_sms\n()\n}\n\"\n)\n# Run a smaller GEMM\nsmall_lhs\n=\ntorch\n.\nrandn\n((\n64\n,\n256\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\nsmall_rhs\n=\ntorch\n.\nrandn\n((\n128\n,\n256\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\nsmall_out\n=\ntorch\n.\nempty\n((\n64\n,\n128\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\nsmall_lhs_fp8\n,\nsmall_lhs_scales\n=\ncast_to_fp8_per_token\n(\nsmall_lhs\n)\nsmall_rhs_fp8\n,\nsmall_rhs_scales\n=\ncast_to_fp8_per_block\n(\nsmall_rhs\n)\ndeep_gemm\n.\ngemm_fp8_fp8_bf16_nt\n(\n(\nsmall_lhs_fp8\n,\nget_col_major_tma_aligned_tensor\n(\nsmall_lhs_scales\n)),\n(\nsmall_rhs_fp8\n,\nsmall_rhs_scales\n),\nsmall_out\n)\n# Restore original setting\ndeep_gemm\n.\nset_num_sms\n(\noriginal_sms\n)\nprint\n(\n\"\n✓ SM configuration demonstrated\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nResource management\n: Control GPU utilization for power efficiency and multi-tenancy.\nPart 16: Debugging and Validation\ndef\nvalidate_fp8_conversion\n(\noriginal\n,\nfp8_data\n,\nscales\n):\n\"\"\"\nCheck if FP8 conversion preserves data accurately\n\"\"\"\n# Reconstruct original from FP8\nif\nfp8_data\n.\ndim\n()\n==\n2\n:\n# Per-token scaling\nm\n,\nn\n=\nfp8_data\n.\nshape\nfp8_view\n=\nfp8_data\n.\nview\n(\nm\n,\n-\n1\n,\n128\n)\nscales_expanded\n=\nscales\n.\nunsqueeze\n(\n2\n)\nreconstructed\n=\nfp8_view\n.\nfloat\n()\n*\nscales_expanded\nreconstructed\n=\nreconstructed\n.\nview\n(\nm\n,\n-\n1\n)[:,\n:\noriginal\n.\nshape\n[\n1\n]]\n# Compare\nabs_error\n=\ntorch\n.\nabs\n(\noriginal\n.\nfloat\n()\n-\nreconstructed\n).\nmax\n().\nitem\n()\nrel_error\n=\nabs_error\n/\ntorch\n.\nabs\n(\noriginal\n.\nfloat\n()).\nmax\n().\nitem\n()\nprint\n(\nf\n\"\nFP8 conversion error:\n{\nabs_error\n:\n.\n6\nf\n}\n(\n{\nrel_error\n*\n100\n:\n.\n3\nf\n}\n%)\n\"\n)\nreturn\nabs_error\n<\n1e-2\n# Reasonable threshold for FP8\n# Validate our conversions\nlhs_valid\n=\nvalidate_fp8_conversion\n(\nlhs\n,\nlhs_fp8\n,\nlhs_scales\n)\nrhs_valid\n=\nvalidate_fp8_conversion\n(\nrhs\n,\nrhs_fp8\n[\n0\n],\nrhs_scales\n[\n0\n])\nprint\n(\nf\n\"\nLHS conversion valid:\n{\nlhs_valid\n}\n\"\n)\nprint\n(\nf\n\"\nRHS conversion valid:\n{\nrhs_valid\n}\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nQuality assurance\n: Always validate FP8 conversions to ensure acceptable precision loss.\nPart 17: Memory Optimization\ndef\nestimate_memory_usage\n(\nshapes\n,\noperation\n=\n\"\ngemm\n\"\n):\n\"\"\"\nEstimate GPU memory usage for DeepGEMM operations\n\"\"\"\nm\n,\nn\n,\nk\n=\nshapes\n# Input tensors\nlhs_fp8\n=\nm\n*\nk\n*\n1\n# FP8\nlhs_scales\n=\nm\n*\n((\nk\n+\n127\n)\n//\n128\n)\n*\n4\n# FP32\nrhs_fp8\n=\nn\n*\nk\n*\n1\n# FP8\nrhs_scales\n=\n((\nn\n+\n127\n)\n//\n128\n)\n*\n((\nk\n+\n127\n)\n//\n128\n)\n*\n4\n# FP32\n# Output\nif\noperation\n==\n\"\ngemm\n\"\n:\noutput\n=\nm\n*\nn\n*\n2\n# BF16\nelse\n:\n# weight_grad\noutput\n=\nm\n*\nn\n*\n4\n# FP32\n# Temporary workspace (estimated)\nworkspace\n=\nmax\n(\nm\n,\nn\n)\n*\n1024\n*\n4\n# Conservative estimate\ntotal\n=\nlhs_fp8\n+\nlhs_scales\n+\nrhs_fp8\n+\nrhs_scales\n+\noutput\n+\nworkspace\nprint\n(\nf\n\"\nMemory usage for\n{\nshapes\n}\n:\n\"\n)\nprint\n(\nf\n\"\nInputs:\n{\n(\nlhs_fp8\n+\nlhs_scales\n+\nrhs_fp8\n+\nrhs_scales\n)\n/\n1024\n**\n2\n:\n.\n1\nf\n}\nMB\n\"\n)\nprint\n(\nf\n\"\nOutput:\n{\noutput\n/\n1024\n**\n2\n:\n.\n1\nf\n}\nMB\n\"\n)\nprint\n(\nf\n\"\nWorkspace:\n{\nworkspace\n/\n1024\n**\n2\n:\n.\n1\nf\n}\nMB\n\"\n)\nprint\n(\nf\n\"\nTotal:\n{\ntotal\n/\n1024\n**\n2\n:\n.\n1\nf\n}\nMB\n\"\n)\nreturn\ntotal\n# Estimate for different problem sizes\nestimate_memory_usage\n((\n1024\n,\n2048\n,\n4096\n),\n\"\ngemm\n\"\n)\nestimate_memory_usage\n((\n2048\n,\n4096\n,\n8192\n),\n\"\nweight_grad\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nCapacity planning\n: Understand memory requirements for different model sizes.\nPart 18: Integration with Training Loops\nclass\nFP8LinearLayer\n:\n\"\"\"\nExample of integrating DeepGEMM into a training loop\n\"\"\"\ndef\n__init__\n(\nself\n,\nin_features\n,\nout_features\n):\nself\n.\nweight\n=\ntorch\n.\nrandn\n((\nout_features\n,\nin_features\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\nself\n.\nweight_grad\n=\ntorch\n.\nzeros_like\n(\nself\n.\nweight\n,\ndtype\n=\ntorch\n.\nfloat\n)\ndef\nforward\n(\nself\n,\nx\n):\n# Convert inputs to FP8\nx_fp8\n,\nx_scales\n=\ncast_to_fp8_per_token\n(\nx\n)\nw_fp8\n,\nw_scales\n=\ncast_to_fp8_per_block\n(\nself\n.\nweight\n)\n# Prepare DeepGEMM inputs\nx_input\n=\n(\nx_fp8\n,\nget_col_major_tma_aligned_tensor\n(\nx_scales\n))\nw_input\n=\n(\nw_fp8\n,\nw_scales\n)\n# Allocate output\noutput\n=\ntorch\n.\nempty\n((\nx\n.\nshape\n[\n0\n],\nself\n.\nweight\n.\nshape\n[\n0\n]),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\n# Forward pass\ndeep_gemm\n.\ngemm_fp8_fp8_bf16_nt\n(\nx_input\n,\nw_input\n,\noutput\n)\nreturn\noutput\ndef\nbackward\n(\nself\n,\nx\n,\ngrad_output\n):\n# Convert to FP8\nx_fp8\n,\nx_scales\n=\ncast_to_fp8_per_token\n(\nx\n)\ngrad_fp8\n,\ngrad_scales\n=\ncast_to_fp8_per_token\n(\ngrad_output\n)\n# Prepare inputs\nx_input\n=\n(\nx_fp8\n,\nget_col_major_tma_aligned_tensor\n(\nx_scales\n))\ngrad_input\n=\n(\ngrad_fp8\n,\nget_col_major_tma_aligned_tensor\n(\ngrad_scales\n))\n# Compute weight gradients: grad_output^T @ x\ndeep_gemm\n.\nwgrad_gemm_fp8_fp8_fp32_nt\n(\ngrad_input\n,\nx_input\n,\nself\n.\nweight_grad\n)\n# Demo usage\nlayer\n=\nFP8LinearLayer\n(\n512\n,\n256\n)\nx\n=\ntorch\n.\nrandn\n((\n128\n,\n512\n),\ndevice\n=\n'\ncuda\n'\n,\ndtype\n=\ntorch\n.\nbfloat16\n)\n# Forward pass\ny\n=\nlayer\n.\nforward\n(\nx\n)\nprint\n(\nf\n\"\nForward output shape:\n{\ny\n.\nshape\n}\n\"\n)\n# Backward pass\ngrad_y\n=\ntorch\n.\nrandn_like\n(\ny\n)\nlayer\n.\nbackward\n(\nx\n,\ngrad_y\n)\nprint\n(\nf\n\"\nWeight grad shape:\n{\nlayer\n.\nweight_grad\n.\nshape\n}\n\"\n)\nprint\n(\n\"\n✓ Training loop integration demonstrated\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nReal-world usage\n: How to integrate DeepGEMM into actual neural network training.\nKey Takeaways\nFP8 = 2x Memory Savings\n: Half the storage with proper scaling\nScaling is Critical\n: Per-token and per-block strategies maintain precision\nTMA Alignment\n: Required for optimal hardware utilization\nGrouped Operations\n: Efficient for MoE and variable-size batches\nJIT Compilation\n: Automatic kernel optimization for each shape\nMemory Layout Matters\n: Column-major scales, contiguous tensors\nFP32 Accumulation\n: Use higher precision for gradients\nPractice Challenge\n# Create an MoE layer with 8 experts\n# Process a batch with variable expert utilization\n# Measure memory savings vs standard implementation\nnum_experts\n=\n8\nexpert_tokens\n=\n[\n64\n,\n128\n,\n96\n,\n112\n,\n88\n,\n144\n,\n72\n,\n104\n]\n# Realistic distribution\nhidden_dim\n=\n2048\n# Your implementation here:\n# 1. Set up grouped GEMM inputs\n# 2. Convert to FP8 \n# 3. Run DeepGEMM\n# 4. Compare with standard PyTorch\n# 5. Measure performance and memory usage\nprint\n(\n\"\nChallenge: Implement efficient MoE with DeepGEMM!\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode",
  "metadata": {
    "title": "DeepGEMM Essentials: High-Performance FP8 Matrix Multiplication - DEV Community",
    "description": "DeepGEMM Essentials: High-Performance FP8 Matrix Multiplication   Google Colab  Master these...",
    "keywords": "software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/vuk_rosic/deepgemm-essentials-high-performance-fp8-matrix-multiplication-i02",
    "og:title": "DeepGEMM Essentials: High-Performance FP8 Matrix Multiplication",
    "og:description": "DeepGEMM Essentials: High-Performance FP8 Matrix Multiplication   Google Colab  Master these...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyktjjx9yuy7lt9pfueb9.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@VukRosic99",
    "twitter:title": "DeepGEMM Essentials: High-Performance FP8 Matrix Multiplication",
    "twitter:description": "DeepGEMM Essentials: High-Performance FP8 Matrix Multiplication   Google Colab  Master these...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyktjjx9yuy7lt9pfueb9.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/183768e5260f/code/body-000.txt",
      "chars": 431,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-001.txt",
      "chars": 476,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-002.txt",
      "chars": 421,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-003.txt",
      "chars": 961,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-004.txt",
      "chars": 949,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-005.txt",
      "chars": 483,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-006.txt",
      "chars": 385,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-007.txt",
      "chars": 854,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-008.txt",
      "chars": 594,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-009.txt",
      "chars": 824,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-010.txt",
      "chars": 770,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-011.txt",
      "chars": 884,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-012.txt",
      "chars": 1073,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-013.txt",
      "chars": 597,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-014.txt",
      "chars": 1006,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-015.txt",
      "chars": 906,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-016.txt",
      "chars": 1002,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-017.txt",
      "chars": 1132,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-018.txt",
      "chars": 1792,
      "language": "text"
    },
    {
      "file": "attachments/183768e5260f/code/body-019.txt",
      "chars": 491,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:24Z",
  "source_url": "https://dev.to/vuk_rosic/deepgemm-essentials-high-performance-fp8-matrix-multiplication-i02",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 20617,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/183768e5260f/pages/source.html",
  "local_html": "attachments/183768e5260f/pages/article.html",
  "body_base_path": "attachments/183768e5260f/pages/",
  "body_markdown_path": "attachments/183768e5260f/pages/article.md"
}