{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "Dive into Deep Learning",
  "title_listed": "Attention Mechanisms and Transformers（章节）",
  "keywords": "Transformer,注意力,自注意力",
  "domain": "大模型算法",
  "content_type": "教程/书籍",
  "is_paid_flag": "否",
  "url": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/index.html",
  "publish_date": "持续更新",
  "corpus_stage": "预训练/SFT",
  "notes": "注意力与 Transformer 章节",
  "id": "30391b9f3347",
  "collected_at": "2026-10-02T16:07:36",
  "title": "11. Attention Mechanisms and Transformers — Dive into Deep Learning 1.0.3 documentation",
  "body_markdown": "Table Of Contents\n11.\nAttention Mechanisms and Transformers\n¶\nThe earliest years of the deep learning boom were driven primarily by\nresults produced using the multilayer perceptron, convolutional network,\nand recurrent network architectures. Remarkably, the model architectures\nthat underpinned many of deep learning’s breakthroughs in the 2010s had\nchanged remarkably little relative to their antecedents despite the\nlapse of nearly 30 years. While plenty of new methodological innovations\nmade their way into most practitioner’s toolkits—ReLU activations,\nresidual layers, batch normalization, dropout, and adaptive learning\nrate schedules come to mind—the core underlying architectures were\nclearly recognizable as scaled-up implementations of classic ideas.\nDespite thousands of papers proposing alternative ideas, models\nresembling classical convolutional neural networks\n(\nSection 7\n) retained\nstate-of-the-art\nstatus in computer\nvision and models resembling Sepp Hochreiter’s original design for the\nLSTM recurrent neural network (\nSection 10.1\n), dominated most\napplications in natural language processing. Arguably, to that point,\nthe rapid emergence of deep learning appeared to be primarily\nattributable to shifts in the available computational resources (thanks\nto innovations in parallel computing with GPUs) and the availability of\nmassive data resources (thanks to cheap storage and Internet services).\nWhile these factors may indeed remain the primary drivers behind this\ntechnology’s increasing power we are also witnessing, at long last, a\nsea change in the landscape of dominant architectures.\nAt the present moment, the dominant models for nearly all natural\nlanguage processing tasks are based on the Transformer architecture.\nGiven any new task in natural language processing, the default\nfirst-pass approach is to grab a large Transformer-based pretrained\nmodel, (e.g., BERT\n(\nDevlin\net al.\n, 2018\n)\n, ELECTRA\n(\nClark\net al.\n, 2020\n)\n, RoBERTa\n(\nLiu\net al.\n, 2019\n)\n, or\nLongformer\n(\nBeltagy\net al.\n, 2020\n)\n) adapting the output layers\nas necessary, and fine-tuning the model on the available data for the\ndownstream task. If you have been paying attention to the last few years\nof breathless news coverage centered on OpenAI’s large language models,\nthen you have been tracking a conversation centered on the GPT-2 and\nGPT-3 Transformer-based models\n(\nBrown\net al.\n, 2020\n,\nRadford\net al.\n, 2019\n)\n. Meanwhile, the\nvision Transformer has emerged as a default model for diverse vision\ntasks, including image recognition, object detection, semantic\nsegmentation, and superresolution\n(\nDosovitskiy\net al.\n, 2021\n,\nLiu\net al.\n, 2021\n)\n. Transformers\nalso showed up as competitive methods for speech recognition\n(\nGulati\net al.\n, 2020\n)\n, reinforcement learning\n(\nChen\net al.\n, 2021\n)\n, and graph neural networks\n(\nDwivedi and Bresson, 2020\n)\n.\nThe core idea behind the Transformer model is the\nattention mechanism\n,\nan innovation that was originally envisioned as an enhancement for\nencoder–decoder RNNs applied to sequence-to-sequence applications, such\nas machine translations\n(\nBahdanau\net al.\n, 2014\n)\n. You might\nrecall that in the first sequence-to-sequence models for machine\ntranslation\n(\nSutskever\net al.\n, 2014\n)\n, the entire input was\ncompressed by the encoder into a single fixed-length vector to be fed\ninto the decoder. The intuition behind attention is that rather than\ncompressing the input, it might be better for the decoder to revisit the\ninput sequence at every step. Moreover, rather than always seeing the\nsame representation of the input, one might imagine that the decoder\nshould selectively focus on particular parts of the input sequence at\nparticular decoding steps. Bahdanau’s attention mechanism provided a\nsimple means by which the decoder could dynamically\nattend\nto\ndifferent parts of the input at each decoding step. The high-level idea\nis that the encoder could produce a representation of length equal to\nthe original input sequence. Then, at decoding time, the decoder can\n(via some control mechanism) receive as input a context vector\nconsisting of a weighted sum of the representations on the input at each\ntime step. Intuitively, the weights determine the extent to which each\nstep’s context “focuses” on each input token, and the key is to make\nthis process for assigning the weights differentiable so that it can be\nlearned along with all of the other neural network parameters.\nInitially, the idea was a remarkably successful enhancement to the\nrecurrent neural networks that already dominated machine translation\napplications. The models performed better than the original\nencoder–decoder sequence-to-sequence architectures. Furthermore,\nresearchers noted that some nice qualitative insights sometimes emerged\nfrom inspecting the pattern of attention weights. In translation tasks,\nattention models often assigned high attention weights to cross-lingual\nsynonyms when generating the corresponding words in the target language.\nFor example, when translating the sentence “my feet hurt” to “j’ai mal\nau pieds”, the neural network might assign high attention weights to the\nrepresentation of “feet” when generating the corresponding French word\n“pieds”. These insights spurred claims that attention models confer\n“interpretability” although what precisely the attention weights\nmean—i.e., how, if at all, they should be\ninterpreted\nremains a hazy\nresearch topic.\nHowever, attention mechanisms soon emerged as more significant concerns,\nbeyond their usefulness as an enhancement for encoder–decoder recurrent\nneural networks and their putative usefulness for picking out salient\ninputs.\nVaswani\net al.\n(\n2017\n)\nproposed the\nTransformer architecture for machine translation, dispensing with\nrecurrent connections altogether, and instead relying on cleverly\narranged attention mechanisms to capture all relationships among input\nand output tokens. The architecture performed remarkably well, and by\n2018 the Transformer began showing up in the majority of\nstate-of-the-art natural language processing systems. Moreover, at the\nsame time, the dominant practice in natural language processing became\nto pretrain large-scale models on enormous generic background corpora to\noptimize some self-supervised pretraining objective, and then to\nfine-tune these models using the available downstream data. The gap\nbetween Transformers and traditional architectures grew especially wide\nwhen applied in this pretraining paradigm, and thus the ascendance of\nTransformers coincided with the ascendence of such large-scale\npretrained models, now sometimes called\nfoundation models\n(\nBommasani\net al.\n, 2021\n)\n.\nIn this chapter, we introduce attention models, starting with the most\nbasic intuitions and the simplest instantiations of the idea. We then\nwork our way up to the Transformer architecture, the vision Transformer,\nand the landscape of modern Transformer-based pretrained models.\n11.1. Queries, Keys, and Values\n11.1.1. Visualization\n11.1.2. Summary\n11.1.3. Exercises\n11.2. Attention Pooling by Similarity\n11.2.1. Kernels and Data\n11.2.2. Attention Pooling via Nadaraya–Watson Regression\n11.2.3. Adapting Attention Pooling\n11.2.4. Summary\n11.2.5. Exercises\n11.3. Attention Scoring Functions\n11.3.1. Dot Product Attention\n11.3.2. Convenience Functions\n11.3.3. Scaled Dot Product Attention\n11.3.4. Additive Attention\n11.3.5. Summary\n11.3.6. Exercises\n11.4. The Bahdanau Attention Mechanism\n11.4.1. Model\n11.4.2. Defining the Decoder with Attention\n11.4.3. Training\n11.4.4. Summary\n11.4.5. Exercises\n11.5. Multi-Head Attention\n11.5.1. Model\n11.5.2. Implementation\n11.5.3. Summary\n11.5.4. Exercises\n11.6. Self-Attention and Positional Encoding\n11.6.1. Self-Attention\n11.6.2. Comparing CNNs, RNNs, and Self-Attention\n11.6.3. Positional Encoding\n11.6.4. Summary\n11.6.5. Exercises\n11.7. The Transformer Architecture\n11.7.1. Model\n11.7.2. Positionwise Feed-Forward Networks\n11.7.3. Residual Connection and Layer Normalization\n11.7.4. Encoder\n11.7.5. Decoder\n11.7.6. Training\n11.7.7. Summary\n11.7.8. Exercises\n11.8. Transformers for Vision\n11.8.1. Model\n11.8.2. Patch Embedding\n11.8.3. Vision Transformer Encoder\n11.8.4. Putting It All Together\n11.8.5. Training\n11.8.6. Summary and Discussion\n11.8.7. Exercises\n11.9. Large-Scale Pretraining with Transformers\n11.9.1. Encoder-Only\n11.9.2. Encoder–Decoder\n11.9.3. Decoder-Only\n11.9.4. Scalability\n11.9.5. Large Language Models\n11.9.6. Summary and Discussion\n11.9.7. Exercises\nPrevious\n10.8. Beam Search\nNext\n11.1. Queries, Keys, and Values",
  "metadata": {
    "title": "11. Attention Mechanisms and Transformers — Dive into Deep Learning 1.0.3 documentation",
    "description": null,
    "keywords": null,
    "author": null
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://www.d2l.ai/_static/logo-with-text.png",
      "bytes": 87175,
      "reused": false,
      "local_path": "attachments/30391b9f3347/images/body-509f124ba61493e64845.png",
      "src": "https://www.d2l.ai/_static/logo-with-text.png",
      "alt": "Dive into Deep Learning",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.d2l.ai/index.html",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_convolutional-neural-networks/index.html",
      "label": "Section 7",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_recurrent-modern/lstm.html",
      "label": "Section 10.1",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_references/zreferences.html",
      "label": "Devlin et al. , 2018",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/queries-keys-values.html",
      "label": "11.1. Queries, Keys, and Values",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/attention-pooling.html",
      "label": "11.2. Attention Pooling by Similarity",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/attention-scoring-functions.html",
      "label": "11.3. Attention Scoring Functions",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/bahdanau-attention.html",
      "label": "11.4. The Bahdanau Attention Mechanism",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/multihead-attention.html",
      "label": "11.5. Multi-Head Attention",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/self-attention-and-positional-encoding.html",
      "label": "11.6. Self-Attention and Positional Encoding",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/transformer.html",
      "label": "11.7. The Transformer Architecture",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/vision-transformer.html",
      "label": "11.8. Transformers for Vision",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/large-pretraining-transformers.html",
      "label": "11.9. Large-Scale Pretraining with Transformers",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_recurrent-modern/beam-search.html",
      "label": "Previous 10.8. Beam Search",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:45Z",
  "source_url": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/index.html",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 8532,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/30391b9f3347/pages/source.html",
  "local_html": "attachments/30391b9f3347/pages/article.html",
  "body_base_path": "attachments/30391b9f3347/pages/",
  "body_markdown_path": "attachments/30391b9f3347/pages/article.md"
}