{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Priyesh Dave",
  "title_listed": "Day 28: Multi-Head Attention",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/priyeshdave6/day-28-multi-head-attention-47g8",
  "publish_date": "2026-09-27",
  "corpus_stage": "",
  "notes": "",
  "id": "ddf6577b32df",
  "collected_at": "2026-10-02T13:48:54",
  "title": "Day 28: Multi-Head Attention - DEV Community",
  "body_markdown": "What is Multi-Head Attention?\nMulti-head attention is a fundamental component of Transformer models. Transformers are now standard for tasks like translation and text summarization. Multi-head attention works by letting the model use several “spotlights,” called attention heads, to focus on different parts of the input at the same time. Each head can highlight different relationships or patterns in the sequence. This setup allows the model to represent more complex information than with a single attention mechanism.\nRecap: Attention Basics\nIn deep learning,\nattention\nis a mechanism that lets a model decide which parts of an input sequence are most relevant when generating each part of the output. When you answer a question about a paragraph, you don’t reread every word equally; you focus on the sentences with the most useful clues. Attention lets a model do something similar—it gives each output position a map showing how much to “look at” each input position, based on learned scores.\nFor example, in a translation system, when generating a word in French, the model can focus on the related English word in the input.\nWhy Multiple Attention Heads?\nA single attention layer is like reading with just one highlighter. You can track one kind of relationship at a time. But language and sequential data have many overlapping patterns, such as grammar, references, and idioms.\nMultiple attention heads\nlet each head “specialize.” One head might pick out short-range patterns, like matching verbs to subjects (“the cat chased the mouse”). Another might track punctuation or sentence structure (“If..., then...” relationships). All heads process the same input but notice different connections.\nThink of a team of detectives. Each detective examines the same evidence but looks for a different clue—fingerprints, timelines, motives. Together, they can solve more than any individual alone.\nHow Multi-Head Attention Works: Step by Step\nLet’s break down the process:\n1. Inputs\nYou start with a batch of input sequences. Each sequence is a list of word or token embeddings—vectors that represent the words.\n2. Linear Projections: Queries, Keys, and Values\nEach input embedding is passed through three trainable linear transformations (simple matrix multiplications):\nQuery (Q):\nRepresents the question or “what we’re looking for.”\nKey (K):\nRepresents what each input token “offers.”\nValue (V):\nThe information available at each token.\nAnalogy: In a database lookup, Query is your search term, Key is a searchable field, and Value is the full record if the Key matches.\n3. Split into Multiple Heads\nInstead of using big Q, K, and V matrices all at once, slice them into\nh\nsmaller versions. Each slice is one head. So if you have 8 heads, you split the embedding into 8 strips—one per head.\n4. Each Head Computes Attention Independently\nEach head does its own attention calculation:\nIt measures how well every Query matches every Key (often by a scaled dot product—multiply and sum the vectors, then scale).\nThe matching scores become weights, showing how much attention to pay to each position.\nThe head uses these weights to combine (blend) the Value vectors, producing a context vector for each head.\n5. Concatenate and Project\nOnce every head has its output, stack the result vectors side by side to re-form the original embedding size. Pass this combined output through one more linear transformation. This mixes the information from all heads.\nKey Terms Recap:\nHead:\nOne independent attention mechanism, focusing on some aspect of the input.\nQuery, Key, Value:\nProjections of the input—each plays a specific role in the attention calculation.\nProjection:\nA trainable linear transformation (matrix multiply) that creates Q, K, or V from the input vectors.\nVisualizing Multi-Head Attention\nPicture a sequence: [A, B, C, D, E, F].\nImagine several stacked rows, one for each head. Head 1 draws arrows from E to D and F—maybe capturing local context. Head 2 links B to A, picking out a name or title. Head 3 draws a wide arc from C to F, perhaps tracking a pronoun linking back to its noun.\nEach head draws different patterns. At the end, their perspectives are joined and distilled into a new embedding for each position in the sequence.\nCode Example: Multi-Head Attention in PyTorch\nBelow is a minimal PyTorch implementation of multi-head attention for small, fixed-size inputs. This code is kept simple to focus on the main idea.\nimport\ntorch\nimport\ntorch.nn.functional\nas\nF\n# Parameters\nbatch_size\n=\n2\nseq_len\n=\n4\n# Sequence length\nembed_dim\n=\n8\n# Embedding size of input tokens\nnum_heads\n=\n2\n# Number of attention heads\nhead_dim\n=\nembed_dim\n//\nnum_heads\n# Example data: batch of sequences, each token is a vector\nx\n=\ntorch\n.\nrandn\n(\nbatch_size\n,\nseq_len\n,\nembed_dim\n)\n# Trainable projection matrices for queries, keys, values\nW_q\n=\ntorch\n.\nrandn\n(\nembed_dim\n,\nembed_dim\n)\nW_k\n=\ntorch\n.\nrandn\n(\nembed_dim\n,\nembed_dim\n)\nW_v\n=\ntorch\n.\nrandn\n(\nembed_dim\n,\nembed_dim\n)\n# Output projection after concatenation\nW_o\n=\ntorch\n.\nrandn\n(\nembed_dim\n,\nembed_dim\n)\ndef\nsplit_heads\n(\ntensor\n,\nnum_heads\n):\n# Reshape last dimension for num_heads, transpose to (batch, heads, seq, head_dim)\nbatch\n,\nseq\n,\nembed\n=\ntensor\n.\nsize\n()\ntensor\n=\ntensor\n.\nview\n(\nbatch\n,\nseq\n,\nnum_heads\n,\nhead_dim\n)\nreturn\ntensor\n.\ntranspose\n(\n1\n,\n2\n)\ndef\ncombine_heads\n(\ntensor\n):\n# Reverse: (batch, heads, seq, head_dim) -> (batch, seq, embed_dim)\nbatch\n,\nnum_heads\n,\nseq\n,\nhead_dim\n=\ntensor\n.\nsize\n()\ntensor\n=\ntensor\n.\ntranspose\n(\n1\n,\n2\n).\ncontiguous\n()\nreturn\ntensor\n.\nview\n(\nbatch\n,\nseq\n,\nnum_heads\n*\nhead_dim\n)\n# 1. Project input vectors for queries, keys, and values\nQ\n=\nx\n@\nW_q\n# (batch, seq, embed_dim)\nK\n=\nx\n@\nW_k\nV\n=\nx\n@\nW_v\n# 2. Split into heads\nQ\n=\nsplit_heads\n(\nQ\n,\nnum_heads\n)\n# (batch, num_heads, seq, head_dim)\nK\n=\nsplit_heads\n(\nK\n,\nnum_heads\n)\nV\n=\nsplit_heads\n(\nV\n,\nnum_heads\n)\n# 3. Scaled dot-product attention for each head\nscores\n=\nQ\n@\nK\n.\ntranspose\n(\n-\n2\n,\n-\n1\n)\n/\n(\nhead_dim\n**\n0.5\n)\n# (batch, num_heads, seq, seq)\nweights\n=\nF\n.\nsoftmax\n(\nscores\n,\ndim\n=-\n1\n)\nheads\n=\nweights\n@\nV\n# (batch, num_heads, seq, head_dim)\n# 4. Concatenate heads and final projection\nconcat\n=\ncombine_heads\n(\nheads\n)\n# (batch, seq, embed_dim)\noutput\n=\nconcat\n@\nW_o\n# (batch, seq, embed_dim)\nprint\n(\n\"\nOutput shape:\n\"\n,\noutput\n.\nshape\n)\n# Output: torch.Size([2, 4, 8])\nEnter fullscreen mode\nExit fullscreen mode\nThis example demonstrates:\nProjecting each token’s vector to query, key, and value vectors.\nDividing them into heads, so each head sees a slice of the embedding.\nEach head running attention in parallel, getting its own view.\nConcatenating all heads and applying an output projection.\nWhen Does Multi-Head Attention Help?\nMulti-head attention makes models better at capturing the complexity and subtlety in sequences. In language, for example, relationships are not just local—words can refer to ideas many tokens away, and multiple patterns overlap.\nWith several heads, models can:\nTrack different kinds of relationships simultaneously.\nBuild separate attention maps for different linguistic or logical functions.\nCombine information across short and long ranges in one pass.\nThis ability is the core reason why Transformers, relying on multi-head attention, became the dominant choice for machine translation, summarization, and general language modeling. It also works well for images, audio, and any task where sequences have several layered relationships.\nKey Takeaways\nMulti-head attention allows the model to focus on different parts of the input sequence simultaneously.\nEach attention head specializes in discovering unique relationships or patterns within the data.\nQueries, Keys, and Values are created via trainable linear projections of the input embeddings.\nOutputs from all heads are concatenated and linearly projected to form the final result.\nThis mechanism enables Transformers to handle complex dependencies in language and other sequential data.\nTry It Yourself\nModify the provided PyTorch code so that each attention head can only attend to a specific subset of sequence positions—for example, restrict the first head to attend only to the first half of the sequence, and the second head to the second half. Run the code and observe how the outputs differ from the original implementation. Write a few sentences describing what changed in the final attention output and what this suggests about the role of attention heads.\nFurther Resources\n🎥\nMulti-Head Attention Explained Visually | Simple Transformer Guide\n🎥\nDay 7 | Transformer Architecture Series | Multi-Head Self-Attention From First Principles\n📄\nHow Transformers Think: Self-Attention and Multi-Head Attention from First Principles\n📄\nSelf-Attention from Scratch: The Core of Every LLM\n📄\nHow to Implement Multi-Head Attention from Scratch in TensorFlow and Keras\nComing up on Day 29:\nPositional Encoding",
  "metadata": {
    "title": "Day 28: Multi-Head Attention - DEV Community",
    "description": "What is Multi-Head Attention?   Multi-head attention is a fundamental component of... Tagged with ai, nlp, genai, 100daysofcode.",
    "keywords": "ai, nlp, genai, 100daysofcode, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/priyeshdave6/day-28-multi-head-attention-47g8",
    "og:title": "Day 28: Multi-Head Attention",
    "og:description": "What is Multi-Head Attention?   Multi-head attention is a fundamental component of...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy5yjer27azf05tx5xnyo.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Day 28: Multi-Head Attention",
    "twitter:description": "What is Multi-Head Attention?   Multi-head attention is a fundamental component of...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy5yjer27azf05tx5xnyo.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/ddf6577b32df/code/body-000.txt",
      "chars": 1940,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=42L1q1Z4Ojc",
      "status": "ok",
      "local_path": "attachments/ddf6577b32df/videos/42L1q1Z4Ojc.mp4",
      "bytes": 19986736,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "643.821000"
    },
    {
      "url": "https://www.youtube.com/watch?v=rKjaI6apojQ",
      "src": "https://www.youtube.com/watch?v=rKjaI6apojQ",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/ddf6577b32df/videos/repair-rKjaI6apojQ.mp4",
      "bytes": 67316951,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "2083.361000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://www.youtube.com/watch?v=42L1q1Z4Ojc",
      "label": "Multi-Head Attention Explained Visually | Simple Transformer Guide",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=rKjaI6apojQ",
      "label": "Day 7 | Transformer Architecture Series | Multi-Head Self-Attention From First Principles",
      "type": "a"
    },
    {
      "href": "https://rishabh-mondal.github.io/blogs/self-attention-multi-head-attention.html",
      "label": "How Transformers Think: Self-Attention and Multi-Head Attention from First Principles",
      "type": "a"
    },
    {
      "href": "https://theaisingularity.org/self-attention-from-scratch/",
      "label": "Self-Attention from Scratch: The Core of Every LLM",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/how-to-implement-multi-head-attention-from-scratch-in-tensorflow-and-keras/",
      "label": "How to Implement Multi-Head Attention from Scratch in TensorFlow and Keras",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:02Z",
  "source_url": "https://dev.to/priyeshdave6/day-28-multi-head-attention-47g8",
  "papers": [],
  "actual_video_files": [
    {
      "local_path": "attachments/ddf6577b32df/videos/42L1q1Z4Ojc.mp4",
      "bytes": 19986736,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "643.821000",
      "status": "ok"
    }
  ],
  "body_chars": 8859,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ddf6577b32df/pages/source.html",
  "local_html": "attachments/ddf6577b32df/pages/article.html",
  "body_base_path": "attachments/ddf6577b32df/pages/",
  "body_markdown_path": "attachments/ddf6577b32df/pages/article.md"
}