{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "zeromathai",
  "title_listed": "How Modern Transformer Blocks Work — From RMSNorm to MoE",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/zeromathai/how-modern-transformer-blocks-work-from-rmsnorm-to-moe-44cc",
  "publish_date": "2026-06-29",
  "corpus_stage": "",
  "notes": "",
  "id": "6599ccf73cfb",
  "collected_at": "2026-10-02T13:50:37",
  "title": "How Modern Transformer Blocks Work — From RMSNorm to MoE - DEV Community",
  "body_markdown": "The original Transformer idea is still alive.\nBut modern LLM blocks are not just the 2017 Transformer copied and scaled.\nThey are engineered for deeper training, longer context, cheaper inference, and larger capacity.\nThat is why components like RMSNorm, GQA, RoPE, SwiGLU, and MoE matter.\nCore Idea\nA modern Transformer block still follows the same basic pattern:\nAttention updates token relationships.\nThe Feed-Forward Network transforms each token representation.\nResidual connections keep information flowing.\nBut modern LLMs changed the details.\nThose details are not cosmetic.\nThey make large-scale training and inference practical.\nThe Key Structure\nA typical modern Transformer block looks like this:\nInput\n→ RMSNorm or Pre-Layer Normalization\n→ Self-Attention with GQA and RoPE\n→ Residual Connection\n→ RMSNorm or Pre-Layer Normalization\n→ Feed-Forward Network with SwiGLU or MoE\n→ Residual Connection\nMore compactly:\nModern Transformer Block = stable normalization + efficient attention + stronger FFN + residual flow\nEach component solves a real scaling problem.\nPre-LN improves deep training stability.\nGQA reduces KV Cache memory.\nRoPE injects position into attention.\nSwiGLU improves FFN expressiveness.\nMoE increases capacity without activating all parameters.\nPseudo-code View\nA simplified modern block looks like this:\ndef transformer_block(x):\n    h = rms_norm(x)\n\n    attn = grouped_query_attention(\n        q=apply_rope(query(h)),\n        k=apply_rope(key(h)),\n        v=value(h)\n    )\n\n    x = x + attn\n\n    h = rms_norm(x)\n\n    ffn = swiglu_ffn(h)\n\n    x = x + ffn\n\n    return x\nEnter fullscreen mode\nExit fullscreen mode\nWith MoE, the FFN part can become:\nh = rms_norm(x)\n\nselected_experts = router(h)\n\nffn = run_top_k_experts(h, selected_experts)\n\nx = x + ffn\nEnter fullscreen mode\nExit fullscreen mode\nThe pattern stays simple.\nNormalize.\nTransform.\nAdd back.\nRepeat.\nConcrete Example\nImagine the model processes this token:\n\"bank\"\nThe attention block helps decide whether “bank” means:\na financial institution\nor the side of a river\nRoPE helps the model understand token order and distance.\nGQA helps attention run with a smaller KV Cache.\nThe FFN then transforms the contextual representation.\nIf the model uses MoE, the router may send this token to experts specialized for finance, geography, or general language.\nThat is the intuition.\nModern Transformer blocks are not just bigger.\nThey are more selective, stable, and hardware-aware.\nPre-LN vs Post-LN\nThe original Transformer commonly used Post-LN.\nPost-LN:\nx = LayerNorm(x + Sublayer(x))\nEnter fullscreen mode\nExit fullscreen mode\nModern LLMs often use Pre-LN.\nPre-LN:\nx = x + Sublayer(LayerNorm(x))\nEnter fullscreen mode\nExit fullscreen mode\nThe difference looks small.\nBut it matters.\nPre-LN normalizes before the sublayer.\nThat helps gradients flow through deep Transformer stacks.\nWhen a model has dozens or hundreds of layers, this becomes critical.\nPre-LN is not just a formatting choice.\nIt is a training stability choice.\nRMSNorm\nRMSNorm is a simpler normalization method.\nLayerNorm recenters and rescales.\nRMSNorm mainly rescales using the root mean square.\nThe RMS is:\nRMS(h) = sqrt((1 / n) * Σ hᵢ²)\nThen the normalized vector is:\nh_norm = h / (RMS(h) + ε) * g\nWhy use it?\nIt keeps activation scale stable.\nIt removes some computation compared with LayerNorm.\nIt works well in large LLMs.\nExample:\nh = [3, 4]\nRMS(h) = sqrt((9 + 16) / 2) ≈ 3.54\nNormalized h ≈ [0.85, 1.13]\nThe key idea:\nRMSNorm stabilizes scale without doing more than necessary.\nAttention Block: GQA + RoPE\nModern attention is often not plain Multi-Head Attention.\nIt usually combines memory-aware attention with positional encoding.\nGrouped-Query Attention reduces KV Cache size.\nRotary Positional Embedding injects position into Query and Key.\nThe attention flow becomes:\nInput\n→ Q, K, V projection\n→ Apply RoPE to Q and K\n→ Share K/V by groups using GQA\n→ Compute attention\n→ Output projection\nThis matters for inference.\nLong-context generation is often limited by KV Cache memory.\nGQA reduces that pressure.\nRoPE keeps position information inside attention without adding a large position table.\nSwiGLU\nThe Feed-Forward Network is not just a simple MLP anymore.\nMany modern LLMs use SwiGLU.\nSwiGLU is a gated activation.\nOne path carries information.\nAnother path controls how much passes through.\nA simplified formula:\nSwiGLU(x) = (W₁x) * Swish(W₂x)\nExample:\nW₁x = 4\nSwish(W₂x) = 0.5\nOutput = 2\nThe gate decides how much information moves forward.\nThat gives the FFN more control than a plain activation.\nMixture of Experts\nMixture of Experts increases model capacity without activating every parameter for every token.\nInstead of one FFN, the model has multiple expert networks.\nA router chooses which experts handle each token.\nExample router output:\nExpert 1 = 0.45\nExpert 2 = 0.19\nExpert 3 = 0.05\nExpert 4 = 0.31\nWith Top-2 routing:\nExpert 1 and Expert 4 are selected.\nOnly those experts run.\nThis is why MoE is called sparse.\nThe model may have many parameters.\nBut each token uses only a small subset.\nDense FFN vs MoE\nDense FFN:\nevery token uses the same FFN\nall FFN parameters are active\nsimpler to train and serve\ncompute grows directly with FFN size\nMoE:\neach token is routed to selected experts\nonly part of the model activates\nincreases total capacity efficiently\nadds routing and load-balancing complexity\nThe key difference:\nDense FFN = same compute path for every token\nMoE = conditional compute path per token\nMoE is powerful.\nBut it is not free.\nIt introduces routing instability, expert imbalance, and distributed communication overhead.\nMulti-Token Prediction\nStandard language modeling predicts one next token.\nAt position t:\npredict token t + 1\nMulti-Token Prediction trains the model to predict multiple future tokens.\nAt position t:\npredict token t + 1, t + 2, t + 3 ...\nThis gives more learning signals from the same representation.\nStandard training:\none position → one supervision signal\nMTP training:\none position → multiple supervision signals\nThis can improve sample efficiency.\nIn some systems, it can also support faster generation ideas.\nNaive vs Modern View\nNaive view:\nTransformer block = attention + FFN\nModern view:\nTransformer block = stable normalization + efficient attention + gated FFN + sparse scaling\nNaive block:\nattention\nffn\nEnter fullscreen mode\nExit fullscreen mode\nModern block:\nrmsnorm\nrope\ngqa\nresidual\nrmsnorm\nswiglu or moe\nresidual\nEnter fullscreen mode\nExit fullscreen mode\nThis matters because modern LLM performance is not just about parameter count.\nIt is about architecture details that make those parameters trainable and deployable.\nImplementation Perspective\nWhen reading modern LLM code, look for these patterns:\nself.input_layernorm = RMSNorm(...)\n\nself.self_attn = Attention(..., rope=True, num_key_value_heads=...)\n\nself.post_attention_layernorm = RMSNorm(...)\n\nself.mlp = SwiGLU(...) or MoE(...)\nEnter fullscreen mode\nExit fullscreen mode\nThe key clue for GQA is:\nnumber of query heads > number of key-value heads\nThe key clue for RoPE is:\nposition is applied to Q and K before attention\nThe key clue for MoE is:\nrouter logits decide which experts run\nThese details tell you what kind of Transformer block you are actually looking at.\nImportant Conditions and Limits\nPre-LN improves stability, but the whole optimization setup still matters.\nRMSNorm is efficient, but it does not replace good initialization or training design.\nGQA reduces KV Cache memory, but may trade off some attention flexibility.\nRoPE works well for long contexts, but very long extrapolation may still need scaling techniques.\nSwiGLU improves FFN behavior, but increases FFN structure complexity.\nMoE increases capacity, but adds routing and system complexity.\nModern Transformer design is a trade-off system.\nEvery upgrade solves one bottleneck and introduces another design choice.\nWhy This Matters Again\nModern LLMs are not just large neural networks.\nThey are carefully engineered stacks.\nIf you understand the block, you can better understand:\nwhy inference needs KV Cache optimization\nwhy RoPE appears in attention code\nwhy RMSNorm replaces LayerNorm\nwhy GQA changes memory usage\nwhy MoE models can be huge but still sparse\nThis is the difference between using LLMs and understanding how they scale.\nTakeaway\nModern Transformer blocks preserve the original Transformer idea.\nBut they upgrade almost every practical detail.\nThe shortest version:\nModern Transformer Block = Pre-LN/RMSNorm + GQA/RoPE Attention + SwiGLU/MoE FFN + Residual Connections\nIf Self-Attention is the core idea, the modern block is the production-grade version of that idea.\nIt is built for depth, context length, inference memory, and scalable capacity.\nDiscussion\nWhen reading modern LLM architecture, which component feels most important to understand first?\nRMSNorm, RoPE, GQA, SwiGLU, or MoE?\nOriginally published at zeromathai.com.\nOriginal article:\nhttps://zeromathai.com/en/modern-transformer-blocks-llm-en/\nGitHub Resources\nAI diagrams, study notes, and visual guides:\nhttps://github.com/zeromathai/zeromathai-ai",
  "metadata": {
    "title": "How Modern Transformer Blocks Work — From RMSNorm to MoE - DEV Community",
    "description": "The original Transformer idea is still alive.  But modern LLM blocks are not just the 2017... Tagged with ai, machinelearning, llm, transformers.",
    "keywords": "ai, machinelearning, llm, transformers, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/zeromathai/how-modern-transformer-blocks-work-from-rmsnorm-to-moe-44cc",
    "og:title": "How Modern Transformer Blocks Work — From RMSNorm to MoE",
    "og:description": "The original Transformer idea is still alive.  But modern LLM blocks are not just the 2017...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3z0bk9x2i29mfb55zo02.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "How Modern Transformer Blocks Work — From RMSNorm to MoE",
    "twitter:description": "The original Transformer idea is still alive.  But modern LLM blocks are not just the 2017...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3z0bk9x2i29mfb55zo02.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/6599ccf73cfb/code/body-000.txt",
      "chars": 265,
      "language": "text"
    },
    {
      "file": "attachments/6599ccf73cfb/code/body-001.txt",
      "chars": 105,
      "language": "text"
    },
    {
      "file": "attachments/6599ccf73cfb/code/body-002.txt",
      "chars": 31,
      "language": "text"
    },
    {
      "file": "attachments/6599ccf73cfb/code/body-003.txt",
      "chars": 31,
      "language": "text"
    },
    {
      "file": "attachments/6599ccf73cfb/code/body-004.txt",
      "chars": 14,
      "language": "text"
    },
    {
      "file": "attachments/6599ccf73cfb/code/body-005.txt",
      "chars": 57,
      "language": "text"
    },
    {
      "file": "attachments/6599ccf73cfb/code/body-006.txt",
      "chars": 187,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://zeromathai.com/en/modern-transformer-blocks-llm-en/",
      "label": "https://zeromathai.com/en/modern-transformer-blocks-llm-en/",
      "type": "a"
    },
    {
      "href": "https://github.com/zeromathai/zeromathai-ai",
      "label": "https://github.com/zeromathai/zeromathai-ai",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:17Z",
  "source_url": "https://dev.to/zeromathai/how-modern-transformer-blocks-work-from-rmsnorm-to-moe-44cc",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 9105,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/6599ccf73cfb/pages/source.html",
  "local_html": "attachments/6599ccf73cfb/pages/article.html",
  "body_base_path": "attachments/6599ccf73cfb/pages/",
  "body_markdown_path": "attachments/6599ccf73cfb/pages/article.md"
}