{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Eugene Yan",
  "title_listed": "Some Intuition on Attention and the Transformer",
  "keywords": "注意力,Transformer,直觉",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://eugeneyan.com/writing/attention/",
  "publish_date": "2023-06",
  "corpus_stage": "预训练/SFT",
  "notes": "注意力机制与Transformer直觉理解",
  "id": "d62e58a7e715",
  "collected_at": "2026-10-02T14:16:48",
  "title": "Some Intuition on Attention and the Transformer",
  "body_markdown": "eugeneyan\nStart Here\nWriting\nSpeaking\nPrototyping\nAbout\nSome Intuition on Attention and the Transformer\n[\ndeeplearning\nllm\n]\n· 8 min read\nChatGPT and other chatbots (e.g., Bard, Claude) have thrust LLMs into the mainstream. As a result, more and more people outside ML and NLP circles are trying to grasp the concept of attention and the Transformer model. Here, we’ll address some questions and try to provide intuition on the Transformer architecture.\nThe intended audience is people who\nhave read the\npaper\nand have a basic understanding of how attention works.\nTo keep it simple, I’ll mostly refer to “words” in a “sentence”. Nonetheless, attention can apply to any generic set of items in a sequence. For example, instead of words, we could have tokens, events, or products. And instead of a sentence, we could have a paragraph, in-session behavior, or purchase history.\nWhat’s the big deal about attention?\nConsider machine translation as an example. Before attention, most translation was done via an encoder-decoder network. The encoder encodes the input sentence (“I love you”) via a recurrent model and the decoder decodes it into another language (“我爱你”).\nEncoding an input sentence into a fixed-size vector for the decoder (\nsource\n)\nVia this approach, the encoder had to cram the entire input into a fixed-size vector which is then passed to the decoder—this single vector had to convey everything about the input sentence! Naturally, this led to an informational bottleneck.\nWith attention, we no longer have to encode input sentences into a single vector.\nInstead, we let the decoder\nattend\nto different words in the input sentence at each step of output generation. This increases the informational capacity, from a single fixed-size vector to the entire sentence (of vectors).\nFurthermore, previous recurrent models had long paths between input and output words. If you had a 50-word sentence, the decoder had to recall information from 50 steps ago for the first word (and that data had to be squeezed into a single vector). As a result, recurrent models had difficulty dealing with long-range dependencies. Attention addressed this by\nletting each step of the decoder see the entire input sentence and decide what words to\nattend\nto.\nThis cut down path length and made it consistent across all steps in the decoder.\nFinally, prior language models leaned heavily on a recurrent approach: To encode a sentence, we start with the first word (\nw1\n) and process it to get the first hidden state (\nh1\n). Then, we input the second word (\nw2\n) with the previous hidden state (\nh1\n) to derive the next hidden state (\nh2\n). And so on. Unfortunately, this process was sequential and prevented parallelization. Attention tackled this by\nreading the entire sentence in one go and computing the representation of each word, based on the sentence, in parallel\n.\nWhat are query, key, and value vectors?\nImagine yourself in a library. You have a specific question (\nquery\n). Books on the shelves have titles on their spines (\nkeys\n) that suggest their content. You compare your question to these titles to decide how relevant each book is, and how much\nattention\nto give each book. Then, you get the information (\nvalue\n) from the relevant books to answer your question.\nIn attention,\nthe\nquery\nrefers to the word we’re computing attention for.\nIn the case of an encoder, the query vector points to the current input word (aka context). For example, if the context was the first word in the input sentence, it would have a query vector\nq1\n.\nThe\nkeys\nrepresent the words in the input sentence.\nThe first word has key vector\nk1\n, the second word has vector\nk2\n, and so on. The key vectors help the model understand how each word relates to the context word. If the first word is the context, we compare the keys to\nq1\n.\nAttention is how much weight the query word\n(e.g.,\nq1\n)\nshould give each word in the sentence\n(e.g.,\nk1\n,\nk2\n, etc). This is computed via a dot product between the query vector and all the key vectors. (A dot product tells us\nhow similar two vectors are\n.) If the dot product between a query-key pair is high, we pay more attention to it. These dot products then go through a\nsoftmax\nwhich makes the attention scores (across all keys) sum to 1.\nEach word is also represented by a\nvalue\nwhich contains the information of that word.\nThese value vectors are weighed by the attention scores that sum to 1. As a result, each context word is now represented by an attention-based weightage of all the words in the sentence, where the most relevant words have higher weight.\nWhat does the encoder and decoder do?\nThe encoder takes a text input, such as a sentence, and returns a sequence of embeddings. These output embeddings can then be used for classification, translation, semantic similarity, etc. Self-attention enables the encoder to weigh the importance of each word and capture both short and long-range dependencies.\nIn contrast, the decoder takes inputs such as a start-of-sentence token and (optional) embeddings from the encoder, and returns probabilities to select the next word. Self-attention enables the decoder to focus on different parts of the output generated so far; cross-attention (aka encoder-decoder attention) helps it attend to the encoder’s output.\nHow does the decoder generate words?\nThe decoder outputs the probability of the next word (i.e., every possible word has an associated probability). Thus, we can generate the next word by greedily picking the word with the highest probability. Alternatively, we can apply\nbeam search\nand keep the top\nn\npredictions, generate the word after next for each of these top\nn\npredictions, and select whichever combination had less error.\nWhy have multiple attention heads?\nMultiple heads lets the model consider multiple words simultaneously.\nBecause we use the softmax function in attention, it amplifies the highest value while squashing the lower ones. As a result, each head tends to focus on a single element.\nConsider the sentence: “The chicken crossed the road carelessly”. The following words are relevant to “crossed” and should be attended to:\nThe “chicken” is the subject doing the crossing.\nThe “road” is the object being crossed.\nThe crossing is done “carelessly”.\nIf we had a single attention head, we might only focus on a single word\n, either “chicken”, “road”, or “crossed”. Multiple heads let us attend to several words.\nIt also provides redundancy\n, where if any single head fails, we have the other attention heads to rely on.\nWhy have multiple attention layers?\nMultiple attention layers builds in redundancy\n(on top of having multiple attention heads). If we only had a single attention layer, that attention layer would have to do a flawless job—this design could be brittle and lead to suboptimal outcomes. We can address this via multiple attention layers, where each one uses the output of the previous layer\nwith the safety net of skip connections\n. Thus, if any single attention layer messed up, the skip connections and downstream layers can mitigate the issue.\nStacking attention layers also broadens the model’s receptive field.\nThe first attention layer produces context vectors by attending to interactions between pairs of words in the input sentence. Then, the second layer produces context vectors based on pairs of pairs, and so on. With more attention layers, the Transformer gains a wider perspective and can attend to multiple interaction levels within the input sentence.\nWhy have skip connections?\nBecause attention acts as a filter, it blocks most information from passing through. As a result, a small change to the inputs of the attention layer may not change the outputs, if the attention score is tiny or zero. This can lead to flat gradients or local optima.\nSkip connections\nhelp dampen the impact of poor attention filtering\n. Even if an input’s attention weight is zero and the input is blocked, skip connections add a copy of that input to the output. This ensures that even small changes to the input can still have noticeable impact on the output.\nFurthermore, skip connections preserve the input sentence:\nThere’s no guarantee that a context word will attend to itself in a transformer. Skip connections ensure this by taking the context word vector and adding it to the output.\n• • •\nFinally, here’s what Andrej Karpathy had to say about Transformers (and Attention).\nThe Transformer is a magnificient neural network architecture because it is a general-purpose differentiable computer. It is simultaneously:\n1) expressive (in the forward pass)\n2) optimizable (via backpropagation+gradient descent)\n3) efficient (high parallelism compute graph)\n— Andrej Karpathy (@karpathy)\nOctober 19, 2022\nReferences\nAttention Is All You Need\nThe Illustrated Transformer\nTransformers From Scratch\nTransformers From Scratch\n(yeap, same title, not an error)\nUnderstanding the Attention Mechanism in Sequence Models\nOG image prompt: “words in a sentence, emphasis on the words, with a pair of glasses, in the style of contrasting tones, artifacts of online culture, innovative page design, complexity theory, bold black and whites, bold color scheme –ar 2:1 –version 5.1”\nIf you found this useful, please cite this write-up as:\nYan, Ziyou. (May 2023). Some Intuition on Attention and the Transformer. eugeneyan.com.\n        https://eugeneyan.com/writing/attention/.\nor\n@article{yan2023attention,\n  title   = {Some Intuition on Attention and the Transformer},\n  author  = {Yan, Ziyou},\n  journal = {eugeneyan.com},\n  year    = {2023},\n  month   = {May},\n  url     = {https://eugeneyan.com/writing/attention/}\n}\nShare on:\nBrowse related tags:\n[\ndeeplearning\nllm\n]\nor\nSearch\n« Open-LLMs - A list of LLMs for Commercial Use\nObsidian-Copilot: An Assistant for Writing & Reflecting »\nJoin\n11,800+\nreaders getting updates on machine learning, RecSys, LLMs, and engineering.\nGet email updates",
  "metadata": {
    "title": "Some Intuition on Attention and the Transformer",
    "description": "What's the big deal, intuition on query-key-value vectors, multiple heads, multiple layers, and more.",
    "keywords": null,
    "author": "Eugene Yan",
    "og:site_name": "eugeneyan.com",
    "og:title": "Some Intuition on Attention and the Transformer",
    "og:type": "article",
    "og:description": "What's the big deal, intuition on query-key-value vectors, multiple heads, multiple layers, and more.",
    "og:url": "https://eugeneyan.com/writing/attention/",
    "og:image": "https://eugeneyan.com/assets/og_image/attention.jpg",
    "twitter:card": "summary_large_image",
    "twitter:domain": "eugeneyan.com",
    "twitter:title": "Some Intuition on Attention and the Transformer",
    "twitter:description": "What's the big deal, intuition on query-key-value vectors, multiple heads, multiple layers, and more.",
    "twitter:image": "https://eugeneyan.com/assets/og_image/attention.jpg",
    "article:published_time": "2023-05-21T00:00:00+00:00",
    "article:author": "https://eugeneyan.com/about/",
    "article:section": "posts",
    "article:tag": "llm"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/d62e58a7e715/code/body-000.txt",
      "chars": 240,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-search.svg",
      "bytes": 425,
      "reused": false,
      "local_path": "attachments/d62e58a7e715/images/body-feeb4bd87b8704a903b2.svg",
      "src": "https://eugeneyan.com/assets/icon-search.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/encoder-decoder.webp",
      "bytes": 8936,
      "reused": false,
      "local_path": "attachments/d62e58a7e715/images/body-c8e7e0090b666e4ffe0e.webp",
      "src": "https://eugeneyan.com/assets/encoder-decoder.webp",
      "alt": "Encoding an input sentence into a fixed-size vector for the decoder",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-twitter.svg",
      "bytes": 892,
      "reused": false,
      "local_path": "attachments/d62e58a7e715/images/body-0083a7581b07e5f40e7b.svg",
      "src": "https://eugeneyan.com/assets/icon-twitter.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "bytes": 559,
      "reused": false,
      "local_path": "attachments/d62e58a7e715/images/body-f7a16c3e46e5838cec4d.svg",
      "src": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/bluesky.svg",
      "bytes": 745,
      "reused": false,
      "local_path": "attachments/d62e58a7e715/images/body-be754e8ecdaf6de94f55.svg",
      "src": "https://eugeneyan.com/assets/bluesky.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-facebook.svg",
      "bytes": 1123,
      "reused": false,
      "local_path": "attachments/d62e58a7e715/images/body-5d55b59733923cf14223.svg",
      "src": "https://eugeneyan.com/assets/icon-facebook.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-mail.svg",
      "bytes": 841,
      "reused": false,
      "local_path": "attachments/d62e58a7e715/images/body-b179147617228dfafb9f.svg",
      "src": "https://eugeneyan.com/assets/icon-mail.svg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://eugeneyan.com/",
      "label": "eugeneyan",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/start-here/",
      "label": "Start Here",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/",
      "label": "Writing",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/speaking/",
      "label": "Speaking",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/prototyping/",
      "label": "Prototyping",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/search/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/deeplearning/",
      "label": "deeplearning",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/llm/",
      "label": "llm",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.03762",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://towardsdatascience.com/understanding-encoder-decoder-sequence-to-sequence-model-679e04af4346",
      "label": "source",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Cosine_similarity",
      "label": "how similar two vectors are",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Softmax_function",
      "label": "softmax",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Beam_search",
      "label": "beam search",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Residual_neural_network",
      "label": "Skip connections",
      "type": "a"
    },
    {
      "href": "https://twitter.com/karpathy/status/1582807367988654081?ref_src=twsrc%5Etfw",
      "label": "October 19, 2022",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/illustrated-transformer/",
      "label": "The Illustrated Transformer",
      "type": "a"
    },
    {
      "href": "https://peterbloem.nl/blog/transformers",
      "label": "Transformers From Scratch",
      "type": "a"
    },
    {
      "href": "https://e2eml.school/transformers.html",
      "label": "Transformers From Scratch",
      "type": "a"
    },
    {
      "href": "https://www.jeremyjordan.me/attention/",
      "label": "Understanding the Attention Mechanism in Sequence Models",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/open-llms/",
      "label": "« Open-LLMs - A list of LLMs for Commercial Use",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/obsidian-copilot/",
      "label": "Obsidian-Copilot: An Assistant for Writing & Reflecting »",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:15Z",
  "source_url": "https://eugeneyan.com/writing/attention/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.03762",
      "bytes": 2215244,
      "reused": true,
      "local_path": "papers/repair/c3ab01e47e614fe3cf33.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 9907,
  "expected_body_images": 7,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/d62e58a7e715/pages/source.html",
  "local_html": "attachments/d62e58a7e715/pages/article.html",
  "body_base_path": "attachments/d62e58a7e715/pages/",
  "body_markdown_path": "attachments/d62e58a7e715/pages/article.md"
}