{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Sebastian Raschka / Ahead of AI",
  "title_listed": "Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs",
  "keywords": "自注意力,多头注意力,因果注意力",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://magazine.sebastianraschka.com/p/understanding-and-coding-self-attention",
  "publish_date": "2024-01",
  "corpus_stage": "预训练/SFT",
  "notes": "注意力机制代码级讲解",
  "id": "3d9578602d43",
  "collected_at": "2026-10-01T20:32:42",
  "title": "Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs",
  "body_markdown": "This article will teach you about self-attention mechanisms used in transformer architectures and large language models (LLMs) such as GPT-4 and Llama. Self-attention and related mechanisms are core components of LLMs, making them a useful topic to understand when working with these models.\nHowever, rather than just discussing the self-attention mechanism, we will code it in Python and PyTorch from the ground up. In my opinion, coding algorithms, models, and techniques from scratch is an excellent way to learn!\nAs a side note, this article is a modernized and extended version of \"\nUnderstanding and Coding the Self-Attention Mechanism of Large Language Models From Scratch\n,\" which I published on my old blog almost exactly a year ago. Since I really enjoy writing (and reading) 'from scratch' articles, I wanted to modernize this article for\nAhead of AI\n.\nAdditionally, this article motivated me to write the book\nBuild a Large Language Model (from Scratch)\n, which is currently in progress. Below is a mental model that summarizes the book and illustrates how the self-attention mechanism fits into the bigger picture.\nAn overview of the topics covered in the\nBuild a Large Language Model (from Scratch)\nbook\nTo keep the length of this article somewhat reasonable, I'll assume you already know about LLMs and you also know about attention mechanisms on a basic level. The goal and focus of this article is to understand how attention mechanisms work via a Python & PyTorch code walkthrough.\nIntroducing Self-Attention\nSince its introduction via the original transformer paper (\nAttention Is All You Need\n), self-attention has become a cornerstone of many state-of-the-art deep learning models, particularly in the field of Natural Language Processing (NLP). Since self-attention is now everywhere, it's important to understand how it works.\nThe original transformer architecture from\nhttps://arxiv.org/abs/1706.03762\nThe concept of \"attention\" in deep learning\nhas its roots in the effort to improve Recurrent Neural Networks (RNNs)\nfor handling longer sequences or sentences. For instance, consider translating a sentence from one language to another. Translating a sentence word-by-word is usually not an option because it ignores the complex grammatical structures and idiomatic expressions unique to each language, leading to inaccurate or nonsensical translations.\nAn incorrect word-by-word translation (top) compared to a correct translation (bottom)\nTo overcome this issue, attention mechanisms were introduced to give access to all sequence elements at each time step. The key is to be selective and determine which words are most important in a specific context.\nIn 2017, the transformer architecture\nintroduced a standalone self-attention mechanism, eliminating the need for RNNs altogether.\n(For brevity, and to keep the article focused on the technical self-attention details, I am keeping this background motivation section brief so that we can focus on the code implementation.)\nA visualization from the “Attention is All You Need” paper (\nhttps://arxiv.org/abs/1706.03762\n) showing how much the word “making” depends or focuses on other words in the input via attention weights (the color intensity is proportional the attention weight value).\nWe can think of self-attention as a mechanism that enhances the information content of an input embedding by including information about the input's context. In other words, the self-attention mechanism enables the model to weigh the importance of different elements in an input sequence and dynamically adjust their influence on the output. This is especially important for language processing tasks, where the meaning of a word can change based on its context within a sentence or document.\nNote that there are many variants of self-attention. A particular focus has been on making self-attention more efficient. However, most papers still implement the original scaled-dot product attention mechanism introduced in the\nAttention Is All You Need paper\nsince self-attention is rarely a computational bottleneck for most companies training large-scale transformers.\nSo, in this article, we focus on the original scaled-dot product attention mechanism (referred to as self-attention), which remains the most popular and most widely used attention mechanism in practice. However, if you are interested in other types of attention mechanisms, check out the\n2020\nEfficient Transformers: A Survey\n, the\n2023\nA Survey on Efficient Training of Transformers\nreview, and the recent\nFlashAttention\nand\nFlashAttention-v2\npapers.\nEmbedding an Input Sentence\nBefore we begin, let's consider an input sentence\n\"Life is short, eat dessert first\"\nthat we want to put through the self-attention mechanism. Similar to other types of modeling approaches for processing text (e.g., using recurrent neural networks or convolutional neural networks), we create a sentence embedding first.\nFor simplicity, here our dictionary\ndc\nis restricted to the words that occur in the input sentence. In a real-world application, we would consider all words in the training dataset (typical vocabulary sizes range between 30k to 50k entries).\nIn:\nsentence = 'Life is short, eat dessert first'\n​\ndc = {s:i for i,s \n      in enumerate(sorted(sentence.replace(',', '').split()))}\n\nprint(dc)\nOut:\n{'Life': 0, 'dessert': 1, 'eat': 2, 'first': 3, 'is': 4, 'short': 5}\nNext, we use this dictionary to assign an integer index to each word:\nIn:\nimport torch\n​\nsentence_int = torch.tensor(\n    [dc[s] for s in sentence.replace(',', '').split()]\n)\nprint(sentence_int)\nOut:\ntensor([0, 4, 5, 2, 1, 3])\nNow, using the integer-vector representation of the input sentence, we can use an embedding layer to encode the inputs into a real-vector embedding. Here, we will use a tiny 3-dimensional embedding such that each input word is represented by a 3-dimensional vector.\nNote that embedding sizes typically range from hundreds to thousands of dimensions. For instance, Llama 2 utilizes embedding sizes of 4,096. The reason we use 3-dimensional embeddings here is purely for illustration purposes. This allows us to examine the individual vectors without filling the entire page with numbers.\nSince the sentence consists of 6 words, this will result in a 6×3-dimensional embedding:\nIn:\nvocab_size = 50_000\n​\ntorch.manual_seed(123)\nembed = torch.nn.Embedding(vocab_size, 3)\nembedded_sentence = embed(sentence_int).detach()\n​\nprint(embedded_sentence)\nprint(embedded_sentence.shape)\nOut:\ntensor([[ 0.3374, -0.1778, -0.3035],\n        [ 0.1794,  1.8951,  0.4954],\n        [ 0.2692, -0.0770, -1.0205],\n        [-0.2196, -0.3792,  0.7671],\n        [-0.5880,  0.3486,  0.6603],\n        [-1.1925,  0.6984, -1.4097]])\ntorch.Size([6, 3])\nDefining the Weight Matrices\nNow, let's discuss the widely utilized self-attention mechanism known as the scaled dot-product attention, which is an integral part of the transformer architecture.\nSelf-attention utilizes three weight matrices, referred to as\nW\nq\n,\nW\nk\n, and\nW\nv\n, which are adjusted as model parameters during training. These matrices serve to project the inputs into\nquery\n,\nkey\n, and\nvalue\ncomponents of the sequence, respectively.\nThe respective query, key and value sequences are obtained via matrix multiplication between the weight matrices\nW\nand the embedded inputs\nx\n:\nQuery sequence:\nq\n(i)\n=\nx\n(i)\nW\nq\nfor\ni\nin sequence\n1 … T\nKey sequence:\nk\n(i)\n=\nx\n(i)\nW\nk\nfor\ni\nin sequence\n1 … T\nValue sequence:\nv\n(i)\n=\nx\n(i)\nW\nv\nfor\ni\nin sequence\n1 … T\nThe index\ni\nrefers to the token index position in the input sequence, which has length\nT\n.\nComputing the query, key, and value vectors via the input x and weights W.\nHere, both\nq\n(i)\nand\nk\n(i)\nare vectors of dimension\nd\nk\n. The projection matrices\nW\nq\nand\nW\nk\nhave a shape of\nd\n×\nd\nk\n, while\nW\nv\nhas the shape\nd\n×\nd\nv\n.\n(It's important to note that\nd\nrepresents the size of each word vector,\nx\n.)\nSince we are computing the dot-product between the query and key vectors, these two vectors have to contain the same number of elements (\nd\nq\n= d\nk\n). In many LLMs, we use the same size for the value vectors such that\nd\nq\n= d\nk\n= d\nv\n. However, the number of elements in the value vector\nv\n(i)\n, which determines the size of the resulting context vector, can be arbitrary.\nSo, for the following code walkthrough, we will set\nd\nq\n= d\nk\n= 2\nand use\nd\nv\n= 4\n, initializing the projection matrices as follows:\nIn:\ntorch.manual_seed(123)\n​\nd = embedded_sentence.shape[1]\n​\nd_q, d_k, d_v = 2, 2, 4\n​\nW_query = torch.nn.Parameter(torch.rand(d, d_q))\nW_key = torch.nn.Parameter(torch.rand(d, d_k))\nW_value = torch.nn.Parameter(torch.rand(d, d_v))\n(Similar to the word embedding vectors earlier, the dimensions\nd\nq\n, d\nk\n, d\nv\nare usually much larger, but we use small numbers here for illustration purposes.)\nAhead of AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.\nSubscribe\nComputing the Unnormalized Attention Weights\nNow, let's suppose we are interested in computing the attention vector for the second input element -- the second input element acts as the query here:\nFor the following sections below, we focus on the second input,\nx\n(2)\nIn code, this looks like as follows:\nIn:\nx_2 = embedded_sentence[1]\nquery_2 = x_2 @ W_query\nkey_2 = x_2 @ W_key\nvalue_2 = x_2 @ W_value\n​\nprint(query_2.shape)\nprint(key_2.shape)\nprint(value_2.shape)\nOut:\ntorch.Size([2])\ntorch.Size([2])\ntorch.Size([4])\nWe can then generalize this to compute the remaining key, and value elements for all inputs as well, since we will need them in the next step when we compute the unnormalized attention weights later:\nIn:\nkeys = embedded_sentence @ W_key\nvalues = embedded_sentence @ W_value\n​\nprint(\"keys.shape:\", keys.shape)\nprint(\"values.shape:\", values.shape)\nOut:\nkeys.shape: torch.Size([6, 2])\nvalues.shape: torch.Size([6, 4])\nNow that we have all the required keys and values, we can proceed to the next step and compute the unnormalized attention weights\nω\n(omega), which are illustrated in the figure below:\nComputing the unnormalized attention weights\nω\n(omega)\nAs illustrated in the figure above, we compute\nω\ni,j\nas the dot product between the query and key sequences,\nω\ni,j\n=\nq\n(i)\nk\n(j)\n.\nFor example, we can compute the unnormalized attention weight for the query and 5th input element (corresponding to index position 4) as follows:\nIn:\nomega_24 = query_2.dot(keys[4])\nprint(omega_24)\n(Note that\nω\nis the symbol for the Greek letter \"omega\", hence the code variable with the same name above.)\nOut:\ntensor(1.2903)\nSince we will need those unnormalized attention weights\nω\nto compute the actual attention weights later, let's compute the\nω\nvalues for all input tokens as illustrated in the previous figure:\nIn:\nomega_2 = query_2 @ keys.T\nprint(omega_2)\nOut:\ntensor([-0.6004,  3.4707, -1.5023,  0.4991,  1.2903, -1.3374])\nComputing the Attention Weights\nThe subsequent step in self-attention is to normalize the unnormalized attention weights,\nω\n, to obtain the normalized attention weights,\nα\n(alpha), by applying the softmax function. Additionally, 1/√{\nd\nk\n} is used to scale\nω\nbefore normalizing it through the softmax function, as shown below:\nComputing the normalized attention weights\nα\nThe scaling by the square-root of\nd\nk\nensures that the Euclidean length of the weight vectors will be approximately in the same magnitude. This helps prevent the attention weights from becoming too small or too large, which could lead to numerical instability or affect the model's ability to converge during training.\nWhy does specifically √\nd\nk\n? The dot product between q and k is a sum of\nd\nk\nindependent terms, each with variance about 1. That means the variance of the raw score grows linearly with\nd\nk\n. By dividing by √\nd\nk\n, we cancel that growth and bring the variance back to about 1.\nIn code, we can implement the computation of the attention weights as follows:\nIn:\nimport torch.nn.functional as F\n​\nattention_weights_2 = F.softmax(omega_2 / d_k**0.5, dim=0)\nprint(attention_weights_2)\nOut:\ntensor([0.0386, 0.6870, 0.0204, 0.0840, 0.1470, 0.0229])\nFinally, the last step is to compute the context vector\nz\n(2)\n, which is an attention-weighted version of our original query input\nx\n(2)\n, including all the other input elements as its context via the attention weights:\nThe attention weights are specific to a certain input element. Here, we chose input element\nx\n(2).\nIn code, this looks like as follows:\nIn:\ncontext_vector_2 = attention_weights_2 @ values\n​\nprint(context_vector_2.shape)\nprint(context_vector_2)\nOut:\ntorch.Size([4])\ntensor([0.5313, 1.3607, 0.7891, 1.3110])\nNote that this output vector has more dimensions (\nd\nv\n= 4\n) than the original input vector (\nd\n= 3\n) since we specified\nd\nv\n> d\nearlier; however, the embedding size choice\nd\nv\nis arbitrary.\nSelf-Attention\nNow, to wrap up the code implementation of the self-attention mechanism in the previous sections above, we can summarize the previous code in a compact\nSelfAttention\nclass:\nIn:\nimport torch.nn as nn\n​\nclass SelfAttention(nn.Module):\n​\n    def __init__(self, d_in, d_out_kq, d_out_v):\n        super().__init__()\n        self.d_out_kq = d_out_kq\n        self.W_query = nn.Parameter(torch.rand(d_in, d_out_kq))\n        self.W_key   = nn.Parameter(torch.rand(d_in, d_out_kq))\n        self.W_value = nn.Parameter(torch.rand(d_in, d_out_v))\n​\n    def forward(self, x):\n        keys = x @ self.W_key\n        queries = x @ self.W_query\n        values = x @ self.W_value\n        \n        attn_scores = queries @ keys.T  # unnormalized attention weights    \n        attn_weights = torch.softmax(\n            attn_scores / self.d_out_kq**0.5, dim=-1\n        )\n        \n        context_vec = attn_weights @ values\n        return context_vec\nFollowing PyTorch conventions, the\nSelfAttention\nclass above initializes the self-attention parameters in the\n__init__\nmethod and computes attention weights and context vectors for all inputs via the\nforward\nmethod. We can use this class as follows:\nIn:\ntorch.manual_seed(123)\n​\n# reduce d_out_v from 4 to 1, because we have 4 heads\nd_in, d_out_kq, d_out_v = 3, 2, 4\n​\nsa = SelfAttention(d_in, d_out_kq, d_out_v)\nprint(sa(embedded_sentence))\nOut:\ntensor([[-0.1564,  0.1028, -0.0763, -0.0764],\n        [ 0.5313,  1.3607,  0.7891,  1.3110],\n        [-0.3542, -0.1234, -0.2627, -0.3706],\n        [ 0.0071,  0.3345,  0.0969,  0.1998],\n        [ 0.1008,  0.4780,  0.2021,  0.3674],\n        [-0.5296, -0.2799, -0.4107, -0.6006]], grad_fn=<MmBackward0>)\nIf you look at the second row, you can see that it matches the values in\ncontext_vector_2\nfrom the previous section exactly:\ntensor([0.5313, 1.3607, 0.7891, 1.3110])\n.\nMulti-Head Attention\nIn the very first figure, at the top of this article (also shown again for convenience below), we saw that transformers use a module called\nmulti-head attention\n.\nThe multi-head attention modules in the original transformer architecture from\nhttps://arxiv.org/abs/1706.03762\nHow does this \"multi-head\" attention module relate to the self-attention mechanism (scaled-dot product attention) we walked through above?\nIn scaled dot-product attention, the input sequence was transformed using three matrices representing the query, key, and value. These three matrices can be considered as a single attention head in the context of multi-head attention. The figure below summarizes this single attention head we covered and implemented previously:\nSummarizing the self-attention mechanism implemented previously\nAs its name implies, multi-head attention involves multiple such heads, each consisting of query, key, and value matrices. This concept is similar to the use of multiple kernels in convolutional neural networks, producing feature maps with multiple output channels.\nMulti-head attention: self-attention with multiple heads\nTo illustrate this in code, we can write a\nMultiHeadAttentionWrapper\nclass for our previous\nSelfAttention\nclass:\nclass MultiHeadAttentionWrapper(nn.Module):\n​\n    def __init__(self, d_in, d_out_kq, d_out_v, num_heads):\n        super().__init__()\n        self.heads = nn.ModuleList(\n            [SelfAttention(d_in, d_out_kq, d_out_v) \n             for _ in range(num_heads)]\n        )\n​\n    def forward(self, x):\n        return torch.cat([head(x) for head in self.heads], dim=-1)\nThe\nd_*\nparameters are the same as before in the\nSelfAttention\nclass -- the only new input parameter here is the number of attention heads:\nd_in\n: Dimension of the input feature vector.\nd_out_kq\n: Dimension for both query and key outputs.\nd_out_v\n: Dimension for value outputs.\nnum_heads\n: Number of attention heads.\nWe initialize the\nSelfAttention\nclass\nnum_heads\ntimes using these input parameters. And we use a PyTorch\nnn.ModuleList\nto store these multiple\nSelfAttention\ninstances.\nThen, the\nforward\npass involves applying each\nSelfAttention\nhead (stored in\nself.heads\n) to the input\nx\nindependently. The results from each head are then concatenated along the last dimension (\ndim=-1\n). Let's see it in action below!\nFirst, let's suppose we have a single Self-Attention head with output dimension 1 to keep it simple for illustration purposes:\nIn:\ntorch.manual_seed(123)\n​\nd_in, d_out_kq, d_out_v = 3, 2, 1\n​\nsa = SelfAttention(d_in, d_out_kq, d_out_v)\nprint(sa(embedded_sentence))\nOut:\ntensor([[-0.0185],\n        [ 0.4003],\n        [-0.1103],\n        [ 0.0668],\n        [ 0.1180],\n        [-0.1827]], grad_fn=<MmBackward0>)\nNow, let's extend this to 4 attention heads:\nIn:\ntorch.manual_seed(123)\n​\nblock_size = embedded_sentence.shape[1]\nmha = MultiHeadAttentionWrapper(\n    d_in, d_out_kq, d_out_v, num_heads=4\n)\n​\ncontext_vecs = mha(embedded_sentence)\n​\nprint(context_vecs)\nprint(\"context_vecs.shape:\", context_vecs.shape)\nOut:\ntensor([[-0.0185,  0.0170,  0.1999, -0.0860],\n        [ 0.4003,  1.7137,  1.3981,  1.0497],\n        [-0.1103, -0.1609,  0.0079, -0.2416],\n        [ 0.0668,  0.3534,  0.2322,  0.1008],\n        [ 0.1180,  0.6949,  0.3157,  0.2807],\n        [-0.1827, -0.2060, -0.2393, -0.3167]], grad_fn=<CatBackward0>)\ncontext_vecs.shape: torch.Size([6, 4])\nBased on the output above, you can see that the single self-attention head created earlier now represents the first column in the output tensor above.\nNotice that the multi-head attention result is a 6×4-dimensional tensor: We have 6 input tokens and 4 self-attention heads, where each self-attention head returns a 1-dimensional output. Previously, in the Self-Attention section, we also produced a 6×4-dimensional tensor. That's because we set the output dimension to 4 instead of 1. In practice, why do we even need multiple attention heads if we can regulate the output embedding size in the\nSelfAttention\nclass itself?\nThe distinction between increasing the output dimension of a single self-attention head and using multiple attention heads lies in how the model processes and learns from the data. While both approaches increase the capacity of the model to represent different features or aspects of the data, they do so in fundamentally different ways.\nFor instance, each attention head in multi-head attention can potentially learn to focus on different parts of the input sequence, capturing various aspects or relationships within the data. This diversity in representation is key to the success of multi-head attention.\nMulti-head attention can also be more efficient, especially in terms of parallel computation. Each head can be processed independently, making it well-suited for modern hardware accelerators like GPUs or TPUs that excel at parallel processing.\nIn short, the use of multiple attention heads is not just about increasing the model's capacity but about enhancing its ability to learn a diverse set of features and relationships within the data. For example, the 7B Llama 2 model uses 32 attention heads.\nCausal Self-Attention\nIn this section, we are adapting the previously discussed self-attention mechanism into a causal self-attention mechanism, specifically for GPT-like (decoder-style) LLMs that are used to generate text. This causal self-attention mechanism  is also often referred to as “masked self-attention”. In the original transformer architecture, it corresponds to the “masked multi-head attention” module — for simplicity, we will look at a single attention head in this section, but the same concept generalizes to multiple heads.\nThe causal self-attention module in the original transformer architecture (via “Attention Is All You Need”,\nhttps://arxiv.org/abs/1706.03762\n)\nCausal self-attention ensures that the  outputs for a certain position in a sequence is based only on the known outputs at previous positions and not on future positions. In simpler terms, it ensures that the prediction for each next word should only depend on the preceding words. To achieve this in GPT-like LLMs, for each token processed, we mask out the future tokens, which come after the current token in the input text.\nThe application of a causal mask to the attention weights for hiding future input tokens in the inputs is illustrated in the figure below.\nTo illustrate and implement causal self-attention, let's work with the unweighted attention scores and attention weights from the previous section. First, we quickly recap the computation of the attention scores from the previous\nSelf-Attention\nsection:\nIn:\ntorch.manual_seed(123)\n​\nd_in, d_out_kq, d_out_v = 3, 2, 4\n​\nW_query = nn.Parameter(torch.rand(d_in, d_out_kq))\nW_key   = nn.Parameter(torch.rand(d_in, d_out_kq))\nW_value = nn.Parameter(torch.rand(d_in, d_out_v))\n​\nx = embedded_sentence\n​\nkeys = x @ W_key\nqueries = x @ W_query\nvalues = x @ W_value\n​\n# attn_scores are the \"omegas\", \n# the unnormalized attention weights\nattn_scores = queries @ keys.T \n​\nprint(attn_scores)\nprint(attn_scores.shape)\nOut:\ntensor([[ 0.0613, -0.3491,  0.1443, -0.0437, -0.1303,  0.1076],\n        [-0.6004,  3.4707, -1.5023,  0.4991,  1.2903, -1.3374],\n        [ 0.2432, -1.3934,  0.5869, -0.1851, -0.5191,  0.4730],\n        [-0.0794,  0.4487, -0.1807,  0.0518,  0.1677, -0.1197],\n        [-0.1510,  0.8626, -0.3597,  0.1112,  0.3216, -0.2787],\n        [ 0.4344, -2.5037,  1.0740, -0.3509, -0.9315,  0.9265]],\n       grad_fn=<MmBackward0>)\ntorch.Size([6, 6])\nSimilar to the\nSelf-Attention\nsection before, the output above is a 6×6 tensor containing these pairwise unnormalized attention weights (also called attention scores) for the 6 input tokens.\nPreviously, we then computed the scaled dot-product attention via the softmax function as follows:\nIn:\nattn_weights = torch.softmax(attn_scores / d_out_kq**0.5, dim=1)\nprint(attn_weights)\nOut:\ntensor([[0.1772, 0.1326, 0.1879, 0.1645, 0.1547, 0.1831],\n        [0.0386, 0.6870, 0.0204, 0.0840, 0.1470, 0.0229],\n        [0.1965, 0.0618, 0.2506, 0.1452, 0.1146, 0.2312],\n        [0.1505, 0.2187, 0.1401, 0.1651, 0.1793, 0.1463],\n        [0.1347, 0.2758, 0.1162, 0.1621, 0.1881, 0.1231],\n        [0.1973, 0.0247, 0.3102, 0.1132, 0.0751, 0.2794]],\n       grad_fn=<SoftmaxBackward0>)\nThe 6×6 output above represents the attention weights, which we also computed in the\nSelf-Attention\nsection before.\nNow, in GPT-like LLMs, we train the model to read and generate one token (or word) at a time, from left to right. If we have a training text sample like \"Life is short eat desert first\" we have the following setup, where the context vectors for the word to the right side of the arrow should only incorporate itself and the previous words:\n\"Life\" → \"is\"\n\"Life is\" → \"short\"\n\"Life is short\" → \"eat\"\n\"Life is short eat\" → \"desert\"\n\"Life is short eat desert\" → \"first\"\nThe simplest way to achieve this setup above is to mask out all future tokens by applying a mask to the attention weight matrix above the diagonal, as illustrated in the figure below. This way, “future” words will not be included when creating the context vectors, which are created as a attention-weighted sum over the inputs.\nAttention weights above the diagonal should be masked out\nIn code, we can achieve this via PyTorch's\ntril\nfunction, which we first use to create a mask of 1's and 0's:\nIn:\nblock_size = attn_scores.shape[0]\nmask_simple = torch.tril(torch.ones(block_size, block_size))\nprint(mask_simple)\nOut:\ntensor([[1., 0., 0., 0., 0., 0.],\n        [1., 1., 0., 0., 0., 0.],\n        [1., 1., 1., 0., 0., 0.],\n        [1., 1., 1., 1., 0., 0.],\n        [1., 1., 1., 1., 1., 0.],\n        [1., 1., 1., 1., 1., 1.]])\nNext, we multiply the attention weights with this mask to zero out all the attention weights above the diagonal:\nIn:\nmasked_simple = attn_weights*mask_simple\nprint(masked_simple)\nOut:\ntensor([[0.1772, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n        [0.0386, 0.6870, 0.0000, 0.0000, 0.0000, 0.0000],\n        [0.1965, 0.0618, 0.2506, 0.0000, 0.0000, 0.0000],\n        [0.1505, 0.2187, 0.1401, 0.1651, 0.0000, 0.0000],\n        [0.1347, 0.2758, 0.1162, 0.1621, 0.1881, 0.0000],\n        [0.1973, 0.0247, 0.3102, 0.1132, 0.0751, 0.2794]],\n       grad_fn=<MulBackward0>)\nWhile the above is one way to mask out future words, notice that the attention weights in each row don't sum to one anymore. To mitigate that, we can normalize the rows such that they sum up to 1 again, which is a standard convention for attention weights:\nIn:\nrow_sums = masked_simple.sum(dim=1, keepdim=True)\nmasked_simple_norm = masked_simple / row_sums\nprint(masked_simple_norm)\nOut:\ntensor([[1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n        [0.0532, 0.9468, 0.0000, 0.0000, 0.0000, 0.0000],\n        [0.3862, 0.1214, 0.4924, 0.0000, 0.0000, 0.0000],\n        [0.2232, 0.3242, 0.2078, 0.2449, 0.0000, 0.0000],\n        [0.1536, 0.3145, 0.1325, 0.1849, 0.2145, 0.0000],\n        [0.1973, 0.0247, 0.3102, 0.1132, 0.0751, 0.2794]],\n       grad_fn=<DivBackward0>)\nAs we can see, the attention weights in each row now sum up to 1.\nNormalizing attention weights in neural networks, such as in transformer models, is advantageous over unnormalized weights for two main reasons. First, normalized attention weights that sum to 1 resemble a probability distribution. This makes it easier to interpret the model's attention to various parts of the input in terms of proportions. Second, by constraining the attention weights to sum to 1, this normalization helps control the scale of the weights and gradients to improve the training dynamics.\nMore efficient masking without renormalization\nIn the causal self-attention procedure we coded above, we first compute the attention scores, then compute the attention weights, mask out attention weights above the diagonal, and lastly renormalize the attention weights. This is summarized in the figure below:\nThe previously implemented causal self-attention procedure\nAlternatively, there is a more efficient way to achieve the same results. In this approach, we take the attention scores and replace the values above the diagonal with negative infinity before the values are input into the softmax function to compute the attention weights. This is summarized in the figure below:\nAn alternative, more efficient approach to implementing causal self-attention\nWe can code up this procedure in PyTorch as follows, starting with masking the attention scores above the diagonal:\nIn:\nmask = torch.triu(torch.ones(block_size, block_size), diagonal=1)\nmasked = attn_scores.masked_fill(mask.bool(), -torch.inf)\nprint(masked)\nThe code above first creates a\nmask\nwith 0s below the diagonal, and 1s above the diagonal. Here,\ntorch.triu\n(\nu\npper\ntri\nangle) retains the elements on and above the main diagonal of a matrix, zeroing out the elements below it, thus preserving the upper triangular portion. In contrast,\ntorch.tril\n(\nl\nower\nt\nriangle) keeps the elements on and below the main diagonal.\nThe\nmasked_fill\nmethod then replaces all the elements above the diagonal via positive mask values (1s) with\n-torch.inf\n, with the results being shown below.\nOut:\ntensor([[ 0.0613,    -inf,    -inf,    -inf,    -inf,    -inf],\n        [-0.6004,  3.4707,    -inf,    -inf,    -inf,    -inf],\n        [ 0.2432, -1.3934,  0.5869,    -inf,    -inf,    -inf],\n        [-0.0794,  0.4487, -0.1807,  0.0518,    -inf,    -inf],\n        [-0.1510,  0.8626, -0.3597,  0.1112,  0.3216,    -inf],\n        [ 0.4344, -2.5037,  1.0740, -0.3509, -0.9315,  0.9265]],\n       grad_fn=<MaskedFillBackward0>)\nThen, all we have to do is to apply the softmax function as usual to obtain the normalized and masked attention weights:\nIn:\nattn_weights = torch.softmax(masked / d_out_kq**0.5, dim=1)\nprint(attn_weights)\nOut:\ntensor([[1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n        [0.0532, 0.9468, 0.0000, 0.0000, 0.0000, 0.0000],\n        [0.3862, 0.1214, 0.4924, 0.0000, 0.0000, 0.0000],\n        [0.2232, 0.3242, 0.2078, 0.2449, 0.0000, 0.0000],\n        [0.1536, 0.3145, 0.1325, 0.1849, 0.2145, 0.0000],\n        [0.1973, 0.0247, 0.3102, 0.1132, 0.0751, 0.2794]],\n       grad_fn=<SoftmaxBackward0>)\nWhy does this work? The softmax function, applied in the last step, converts the input values into a probability distribution. When\n-inf\nis present in the inputs, softmax effectively treats them as zero probability. This is because\ne^(-inf)\napproaches 0, and thus these positions contribute nothing to the output probabilities.\nConclusion\nIn this article, we explored the inner workings of self-attention through a step-by-step coding approach. Using this as a foundation, we then looked into multi-head attention, a fundamental component of large language transformers.\nWe then also coded cross-attention, a variant of self-attention that is particularly effective when applied between two distinct sequences. And lastly, we coded causal self-attention, a concept crucial for generating coherent and contextually appropriate sequences in decoder-style LLMs such as GPT and Llama.\nBy coding these complex mechanisms from scratch, you hopefully gained a good understanding of the inner workings of the self-attention mechanism used in transformers and LLMs.\n(Note that the code presented in this article is intended for illustrative purposes. If you plan to implement self-attention for training LLMs, I recommend considering optimized implementations like\nFlash Attention\n, which reduce memory footprint and computational load.)\nBonus Topic: Cross-Attention\nIn the code walkthrough above, the sections on Self-Attention and Causal-Attention, we set\nd_q = d_k = 2\nand\nd_v = 4\n. In other words, we used the same dimensions for query and key sequences. While the value matrix\nW_v\nis often chosen to have the same dimension as the query and key matrices (such as in PyTorch's\nMultiHeadAttention\nclass), we can select an arbitrary number size for the value dimensions…",
  "metadata": {
    "title": "Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs",
    "description": "This article codes the self-attention mechanisms used in transformer architectures and large language models (LLMs) such as GPT-4 and Llama from scratch in PyTorch.",
    "keywords": null,
    "author": "Sebastian Raschka, PhD",
    "og:type": "article",
    "og:title": "Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs",
    "og:description": "This article will teach you about self-attention mechanisms used in transformer architectures and large language models (LLMs) such as GPT-4 and Llama.",
    "og:image": "https://substackcdn.com/image/fetch/$s_!3NS4!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69bfee26-ea3b-42a6-8a1a-6b8187852082_738x564.png",
    "og:url": "https://magazine.sebastianraschka.com/p/understanding-and-coding-self-attention",
    "twitter:title": "Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs",
    "twitter:description": "This article will teach you about self-attention mechanisms used in transformer architectures and large language models (LLMs) such as GPT-4 and Llama.",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!v80X!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fsebastianraschka.substack.com%2Fapi%2Fv1%2Fpost_preview%2F140464659%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2026-03-09T17:54:09.223Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/3d9578602d43/code/body-000.txt",
      "chars": 140,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-001.txt",
      "chars": 68,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-002.txt",
      "chars": 120,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-003.txt",
      "chars": 26,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-004.txt",
      "chars": 193,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-005.txt",
      "chars": 241,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-006.txt",
      "chars": 228,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-007.txt",
      "chars": 157,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-008.txt",
      "chars": 47,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-009.txt",
      "chars": 141,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-010.txt",
      "chars": 63,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-011.txt",
      "chars": 47,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-012.txt",
      "chars": 14,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-013.txt",
      "chars": 41,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-014.txt",
      "chars": 62,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-015.txt",
      "chars": 119,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-016.txt",
      "chars": 56,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-017.txt",
      "chars": 103,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-018.txt",
      "chars": 56,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-019.txt",
      "chars": 751,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-020.txt",
      "chars": 187,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-021.txt",
      "chars": 299,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-022.txt",
      "chars": 366,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-023.txt",
      "chars": 133,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-024.txt",
      "chars": 137,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-025.txt",
      "chars": 251,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-026.txt",
      "chars": 339,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-027.txt",
      "chars": 448,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-028.txt",
      "chars": 433,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-029.txt",
      "chars": 84,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-030.txt",
      "chars": 383,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-031.txt",
      "chars": 113,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-032.txt",
      "chars": 204,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-033.txt",
      "chars": 61,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-034.txt",
      "chars": 379,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-035.txt",
      "chars": 121,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-036.txt",
      "chars": 379,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-037.txt",
      "chars": 137,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-038.txt",
      "chars": 422,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-039.txt",
      "chars": 79,
      "language": "text"
    },
    {
      "file": "attachments/3d9578602d43/code/body-040.txt",
      "chars": 385,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!zkNz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F924a2d16-e8c2-4acb-84ba-bf95f2df1b31_1342x440.png",
      "bytes": 75907,
      "reused": true,
      "local_path": "attachments/3d9578602d43/images/014.png",
      "src": "https://substackcdn.com/image/fetch/$s_!zkNz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F924a2d16-e8c2-4acb-84ba-bf95f2df1b31_1342x440.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!BtVj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97567e7b-f8b9-4dea-a678-162378609a75_1304x1150.png",
      "bytes": 114316,
      "reused": true,
      "local_path": "attachments/3d9578602d43/images/025.png",
      "src": "https://substackcdn.com/image/fetch/$s_!BtVj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97567e7b-f8b9-4dea-a678-162378609a75_1304x1150.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!uRaO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d632b81-5bd6-432a-a456-37f20788be20_1180x614.png",
      "bytes": 94426,
      "reused": true,
      "local_path": "attachments/3d9578602d43/images/036.png",
      "src": "https://substackcdn.com/image/fetch/$s_!uRaO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d632b81-5bd6-432a-a456-37f20788be20_1180x614.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!sSRX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49828e20-cd55-47bb-84a3-59effec1be79_3140x4251.png",
      "bytes": 313892,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-75c054058173b7f7e851.png",
      "src": "https://substackcdn.com/image/fetch/$s_!sSRX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49828e20-cd55-47bb-84a3-59effec1be79_3140x4251.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!3woZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecef3e00-4c0e-4c7a-9a9f-42ac4e4ada69_366x786.png",
      "bytes": 43453,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-7219fd58635be09692d7.png",
      "src": "https://substackcdn.com/image/fetch/$s_!3woZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecef3e00-4c0e-4c7a-9a9f-42ac4e4ada69_366x786.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!qmcs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9774001-ea9d-48bf-9857-3c911b0a279d_588x962.png",
      "bytes": 67934,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-9eb5a2a48d92117d6fdb.png",
      "src": "https://substackcdn.com/image/fetch/$s_!qmcs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9774001-ea9d-48bf-9857-3c911b0a279d_588x962.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!7mUN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbaf9e308-223b-429e-8527-a7b868003e8c_814x912.png",
      "bytes": 98085,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-4215b1b44c82333767f7.png",
      "src": "https://substackcdn.com/image/fetch/$s_!7mUN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbaf9e308-223b-429e-8527-a7b868003e8c_814x912.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!rgXp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42da287a-18e8-45c7-860c-46e8a3a534fc_1400x798.png",
      "bytes": 97544,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-c93a23f922407d509508.png",
      "src": "https://substackcdn.com/image/fetch/$s_!rgXp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42da287a-18e8-45c7-860c-46e8a3a534fc_1400x798.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!GpWw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e1dcdeb-e096-4ff9-bdf9-3338e4efa4b4_1916x1048.png",
      "bytes": 84373,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-f5414c8037c8413b71d0.png",
      "src": "https://substackcdn.com/image/fetch/$s_!GpWw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e1dcdeb-e096-4ff9-bdf9-3338e4efa4b4_1916x1048.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!vVWg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57269a7a-8ecc-4ed6-b6d7-79a9982dd776_918x1152.png",
      "bytes": 81556,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-e5d460ed3d25c796eb25.png",
      "src": "https://substackcdn.com/image/fetch/$s_!vVWg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57269a7a-8ecc-4ed6-b6d7-79a9982dd776_918x1152.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!sfR2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7651e279-01ed-4316-91dc-d6be8ef4e947_1836x1116.png",
      "bytes": 103292,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-1542163e82f5b4842910.png",
      "src": "https://substackcdn.com/image/fetch/$s_!sfR2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7651e279-01ed-4316-91dc-d6be8ef4e947_1836x1116.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!57T6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff406d55e-990a-4e3b-be82-d966eb74a3e7_1766x1154.png",
      "bytes": 123946,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-c184939554fc56853c26.png",
      "src": "https://substackcdn.com/image/fetch/$s_!57T6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff406d55e-990a-4e3b-be82-d966eb74a3e7_1766x1154.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!vQYr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc51bfe11-c2cf-4ce5-95d4-4f8a57eac997_1026x1148.png",
      "bytes": 93251,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-1767aabf00d59619e1b8.png",
      "src": "https://substackcdn.com/image/fetch/$s_!vQYr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc51bfe11-c2cf-4ce5-95d4-4f8a57eac997_1026x1148.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!W-5R!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7dd5ec8-4912-4e0d-8265-b335c08d2958_1760x1126.png",
      "bytes": 158593,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-930d5486660f747d76ac.png",
      "src": "https://substackcdn.com/image/fetch/$s_!W-5R!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7dd5ec8-4912-4e0d-8265-b335c08d2958_1760x1126.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!rOs7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1317a05-3542-4158-94bf-085109a5793a_1220x702.png",
      "bytes": 228832,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-0e9f10d44057fbb5fcd1.png",
      "src": "https://substackcdn.com/image/fetch/$s_!rOs7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1317a05-3542-4158-94bf-085109a5793a_1220x702.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!pRz7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fcce180-cda2-4094-bb7c-32ea12e60c9a_1468x274.png",
      "bytes": 31968,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-d2a966849e45b143f581.png",
      "src": "https://substackcdn.com/image/fetch/$s_!pRz7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7fcce180-cda2-4094-bb7c-32ea12e60c9a_1468x274.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!vh8x!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa94e36c7-ec67-43d6-96c0-7f9781f402b5_1054x270.png",
      "bytes": 35681,
      "reused": false,
      "local_path": "attachments/3d9578602d43/images/body-cabf942c66257195cc46.png",
      "src": "https://substackcdn.com/image/fetch/$s_!vh8x!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa94e36c7-ec67-43d6-96c0-7f9781f402b5_1054x270.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html",
      "label": "Understanding and Coding the Self-Attention Mechanism of Large Language Models From Scratch",
      "type": "a"
    },
    {
      "href": "http://mng.bz/amjo",
      "label": "Build a Large Language Model (from Scratch)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.03762",
      "label": "Attention Is All You Need",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!BtVj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97567e7b-f8b9-4dea-a678-162378609a75_1304x1150.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1409.0473",
      "label": "has its roots in the effort to improve Recurrent Neural Networks (RNNs)",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!sSRX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49828e20-cd55-47bb-84a3-59effec1be79_3140x4251.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.06732",
      "label": "2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2302.01107",
      "label": "2023",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2205.14135",
      "label": "FlashAttention",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2307.08691",
      "label": "FlashAttention-v2",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!3woZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecef3e00-4c0e-4c7a-9a9f-42ac4e4ada69_366x786.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!qmcs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9774001-ea9d-48bf-9857-3c911b0a279d_588x962.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!7mUN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbaf9e308-223b-429e-8527-a7b868003e8c_814x912.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!rgXp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F42da287a-18e8-45c7-860c-46e8a3a534fc_1400x798.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!GpWw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e1dcdeb-e096-4ff9-bdf9-3338e4efa4b4_1916x1048.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!vVWg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57269a7a-8ecc-4ed6-b6d7-79a9982dd776_918x1152.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!sfR2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7651e279-01ed-4316-91dc-d6be8ef4e947_1836x1116.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!57T6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff406d55e-990a-4e3b-be82-d966eb74a3e7_1766x1154.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!vQYr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc51bfe11-c2cf-4ce5-95d4-4f8a57eac997_1026x1148.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!W-5R!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb7dd5ec8-4912-4e0d-8265-b335c08d2958_1760x1126.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!rOs7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1317a05-3542-4158-94bf-085109a5793a_1220x702.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/docs/stable/generated/torch.tril.html",
      "label": "tril",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html",
      "label": "MultiHeadAttention",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:24Z",
  "source_url": "https://magazine.sebastianraschka.com/p/understanding-and-coding-self-attention",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.03762",
      "bytes": 2215244,
      "reused": true,
      "local_path": "papers/repair/c3ab01e47e614fe3cf33.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1409.0473",
      "bytes": 444482,
      "reused": true,
      "local_path": "papers/repair/b6909588ff7b9ebf375f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.06732",
      "bytes": 847824,
      "reused": false,
      "local_path": "papers/repair/5442c18a5d3ec7c8eca5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2302.01107",
      "bytes": 366251,
      "reused": false,
      "local_path": "papers/repair/f08b9d9967e83d2d11a7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2205.14135",
      "bytes": 2630825,
      "reused": false,
      "local_path": "papers/repair/bc0de6a7863ada2c8edb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2307.08691",
      "bytes": 1660578,
      "reused": false,
      "local_path": "papers/repair/7404c00d30e9fc79c6c8.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 30643,
  "expected_body_images": 17,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/3d9578602d43/pages/source.html",
  "local_html": "attachments/3d9578602d43/pages/article.html",
  "body_base_path": "attachments/3d9578602d43/pages/",
  "body_markdown_path": "attachments/3d9578602d43/pages/article.md"
}