{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "zeromathai",
  "title_listed": "Why Multi-Head Attention Needs Position, Residuals, and Normalization",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/zeromathai/why-multi-head-attention-needs-position-residuals-and-normalization-15nj",
  "publish_date": "2026-06-22",
  "corpus_stage": "",
  "notes": "",
  "id": "56a2112d37bb",
  "collected_at": "2026-10-02T13:50:43",
  "title": "Why Multi-Head Attention Needs Position, Residuals, and Normalization - DEV Community",
  "body_markdown": "Self-Attention is powerful.\nBut by itself, it has three problems.\nIt needs multiple views, it needs word order, and it needs stable training.\nThat is why Multi-Head Attention, Positional Encoding, and Add & Norm exist.\nCore Idea\nA Transformer block is not just attention.\nAttention computes token relationships.\nMulti-Head Attention makes those relationships richer.\nPositional Encoding tells the model where tokens are.\nAdd & Norm keeps deep Transformer blocks trainable.\nThis matters because modern LLMs are deep.\nWithout these support structures, attention alone is not enough.\nThe Key Structure\nA simplified Transformer encoder block looks like this:\nInput\n→ Positional Information\n→ Multi-Head Attention\n→ Add & Norm\n→ Feed-Forward Network\n→ Add & Norm\n→ Output\nMore compactly:\nTransformer Block = attention + position + residual flow + normalization\nEach part solves a specific problem.\nMulti-Head Attention solves the “single view” problem.\nPositional Encoding solves the “no order” problem.\nAdd & Norm solves the “deep training stability” problem.\nImplementation View\nAt a high level, the block works like this:\ntokens = tokenize(text)\n\nx = embedding(tokens)\n\nx = x + positional_encoding\n\nattention_output = multi_head_attention(x)\n\nx = layer_norm(x + attention_output)\n\nffn_output = feed_forward(x)\n\noutput = layer_norm(x + ffn_output)\nEnter fullscreen mode\nExit fullscreen mode\nIn modern Pre-LN style, the order often changes:\nattention_output = multi_head_attention(layer_norm(x))\n\nx = x + attention_output\n\nffn_output = feed_forward(layer_norm(x))\n\noutput = x + ffn_output\nEnter fullscreen mode\nExit fullscreen mode\nThe idea is the same.\nKeep the original signal flowing.\nNormalize activations.\nLet attention and FFN update the representation.\nMulti-Head Attention\nSingle attention gives one relationship map.\nBut language has many relationship types.\nA token may need to track:\nnearby words\nsubject-verb structure\nsemantic similarity\nlong-distance references\ncoreference\nOne attention head cannot easily capture all of these at once.\nMulti-Head Attention fixes this by running several attention heads in parallel.\nEach head gets its own learned Q, K, and V projections.\nSo each head can learn a different representation subspace.\nThe formula is:\nMultiHead(Q, K, V) = Concat(head₁, ..., headₕ)Wᴼ\nIn plain English:\nRun attention multiple ways.\nConcatenate the results.\nProject them back into one vector space.\nConcrete Example\nTake this sentence:\nThe animal did not cross the street because it was tired.\nWhat does “it” refer to?\nOne attention head may focus on “animal.”\nAnother may focus on “tired.”\nAnother may track the structure around “because.”\nThis is useful because the model does not need one attention map to explain everything.\nDifferent heads can specialize.\nThat is why Multi-Head Attention matters.\nIt gives the model multiple ways to read the same sentence.\nSingle-Head vs Multi-Head Attention\nSingle-head attention:\nuses one attention distribution\nsees token relationships from one perspective\nis simpler\ncan mix different patterns together\nMulti-head attention:\nuses multiple attention distributions\nviews tokens through different learned projections\ncaptures diverse relationships\nrecombines the results afterward\nThe key difference:\nSingle-head = one view of context\nMulti-head = multiple views of context\nThis is not just repeated computation.\nIt is structured parallel interpretation.\nWhy Positional Encoding Is Needed\nSelf-Attention compares tokens at the same time.\nThat is great for parallelism.\nBut it creates a problem.\nAttention alone does not know token order.\nConsider:\ndog bites man\nman bites dog\nSame words.\nDifferent meaning.\nWithout position information, the model does not naturally know which token came first.\nSo Transformers inject position into token representations.\nThe basic structure is:\nInput Representation = Token Embedding + Positional Encoding\nThis matters because word order changes meaning.\nA language model must know both:\nwhat the token is\nwhere the token is\nSinusoidal Positional Encoding\nThe original Transformer used fixed sine and cosine patterns.\nEven dimensions use sine.\nOdd dimensions use cosine.\nThe idea is:\ndifferent positions get different wave patterns.\nA simplified view:\nPE(pos, 2i) = sin(pos / 10000^(2i / d_model))\nPE(pos, 2i + 1) = cos(pos / 10000^(2i / d_model))\nThis is not just a position ID.\nIt creates smooth position signals across dimensions.\nThe model can use these signals to reason about position and distance.\nAPE, RPE, and RoPE\nThere are several ways to inject position.\nAbsolute Positional Embedding:\nassigns a position vector to each absolute index\nposition 1 has one vector\nposition 2 has another vector\nRelative Positional Embedding:\nfocuses on distance between tokens\nuseful when relative position matters more than absolute index\nRotary Positional Embedding:\nrotates Query and Key vectors using position\nmakes relative position work naturally inside attention\ncommonly used in modern LLMs\nThe shared goal is simple:\nGive attention a way to understand order.\nAdd & Norm\nA Transformer block also needs stability.\nThat is where Add & Norm comes in.\nAdd means residual connection.\nNorm means layer normalization.\nThe classic Post-LN formula is:\nOutput = LayerNorm(x + Sublayer(x))\nThe residual connection preserves the original input.\nThe sublayer adds a learned update.\nLayerNorm keeps the representation stable.\nThis is important because Transformers stack many layers.\nWithout residual paths, information can degrade.\nWithout normalization, training can become unstable.\nResidual Connection Intuition\nA sublayer should not have to rebuild everything from scratch.\nIt should only need to learn an update.\nInstead of:\nnew_output = sublayer(x)\nUse:\nnew_output = x + sublayer(x)\nThis is a huge difference.\nThe original representation can pass forward directly.\nThe sublayer only adds useful changes.\nThat makes deep networks easier to train.\nLayer Normalization Intuition\nLayerNorm normalizes each token representation.\nIt works across the feature dimensions of a token.\nNot across the sequence.\nThat means each token vector is stabilized independently.\nIn practice, this helps keep activation values in a manageable range.\nThis matters when many Transformer blocks are stacked.\nSmall instability in one layer can grow across dozens or hundreds of layers.\nPre-LN vs Post-LN\nPost-LN:\nx = LayerNorm(x + Sublayer(x))\nEnter fullscreen mode\nExit fullscreen mode\nPre-LN:\nx = x + Sublayer(LayerNorm(x))\nEnter fullscreen mode\nExit fullscreen mode\nThe original Transformer used Post-LN.\nMany modern large Transformers use Pre-LN.\nWhy?\nPre-LN often improves training stability in deep models.\nThe difference is placement.\nPost-LN normalizes after the residual addition.\nPre-LN normalizes before the sublayer.\nBoth use residual connections.\nBut their training behavior can be different.\nNaive vs Practical View\nNaive view:\nTransformer block = attention layer\nPractical view:\nTransformer block = attention + position + residuals + normalization + FFN\nNaive implementation mindset:\nrun attention\nreturn output\nEnter fullscreen mode\nExit fullscreen mode\nPractical implementation mindset:\nadd position\nrun multi-head attention\npreserve input through residuals\nnormalize representations\napply feed-forward updates\nrepeat safely across many layers\nEnter fullscreen mode\nExit fullscreen mode\nThis is why implementation details matter.\nThe architecture works because these parts support each other.\nImportant Conditions and Limits\nMulti-Head Attention is powerful, but more heads are not always better.\nToo many heads can increase cost.\nSome heads may become redundant.\nPositional Encoding is necessary because attention is order-agnostic by default.\nBut different positional methods behave differently in long-context settings.\nAdd & Norm improves stability.\nBut the exact Pre-LN or Post-LN choice affects optimization.\nSo these are not decorative components.\nThey are architectural decisions.\nTakeaway\nMulti-Head Attention gives the model multiple views of token relationships.\nPositional Encoding gives attention a sense of order.\nAdd & Norm keeps deep Transformer blocks stable.\nThe shortest version is:\nTransformer Block = multi-view attention + position signal + stable residual updates\nIf Self-Attention is the engine, these components are the systems that make the engine usable at scale.\nDiscussion\nWhen reading Transformer architecture, which part feels most important to understand first?\nMulti-Head Attention, Positional Encoding, or Add & Norm?\nOriginally published at zeromathai.com.\nOriginal article:\nhttps://zeromathai.com/en/multi-head-attention-positional-encoding-add-norm-en/\nGitHub Resources\nAI diagrams, study notes, and visual guides:\nhttps://github.com/zeromathai/zeromathai-ai",
  "metadata": {
    "title": "Why Multi-Head Attention Needs Position, Residuals, and Normalization - DEV Community",
    "description": "Self-Attention is powerful.  But by itself, it has three problems.  It needs multiple views, it needs... Tagged with ai, machinelearning, llm, deeplearning.",
    "keywords": "ai, machinelearning, llm, deeplearning, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/zeromathai/why-multi-head-attention-needs-position-residuals-and-normalization-15nj",
    "og:title": "Why Multi-Head Attention Needs Position, Residuals, and Normalization",
    "og:description": "Self-Attention is powerful.  But by itself, it has three problems.  It needs multiple views, it needs...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzl17gnzc8eai4tf3a0jq.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Why Multi-Head Attention Needs Position, Residuals, and Normalization",
    "twitter:description": "Self-Attention is powerful.  But by itself, it has three problems.  It needs multiple views, it needs...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzl17gnzc8eai4tf3a0jq.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/56a2112d37bb/code/body-000.txt",
      "chars": 225,
      "language": "text"
    },
    {
      "file": "attachments/56a2112d37bb/code/body-001.txt",
      "chars": 148,
      "language": "text"
    },
    {
      "file": "attachments/56a2112d37bb/code/body-002.txt",
      "chars": 31,
      "language": "text"
    },
    {
      "file": "attachments/56a2112d37bb/code/body-003.txt",
      "chars": 31,
      "language": "text"
    },
    {
      "file": "attachments/56a2112d37bb/code/body-004.txt",
      "chars": 28,
      "language": "text"
    },
    {
      "file": "attachments/56a2112d37bb/code/body-005.txt",
      "chars": 157,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://zeromathai.com/en/multi-head-attention-positional-encoding-add-norm-en/",
      "label": "https://zeromathai.com/en/multi-head-attention-positional-encoding-add-norm-en/",
      "type": "a"
    },
    {
      "href": "https://github.com/zeromathai/zeromathai-ai",
      "label": "https://github.com/zeromathai/zeromathai-ai",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:18Z",
  "source_url": "https://dev.to/zeromathai/why-multi-head-attention-needs-position-residuals-and-normalization-15nj",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 8776,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/56a2112d37bb/pages/source.html",
  "local_html": "attachments/56a2112d37bb/pages/article.html",
  "body_base_path": "attachments/56a2112d37bb/pages/",
  "body_markdown_path": "attachments/56a2112d37bb/pages/article.md"
}