{ "source_file": "0930", "source_platform": "个人博客", "author": "Lilian Weng / Lil'Log", "title_listed": "Attention? Attention!", "keywords": "注意力机制,自注意力,多头", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/", "publish_date": "2018-06", "corpus_stage": "预训练/SFT", "notes": "注意力机制经典综述", "id": "1c9291c74af3", "collected_at": "2026-10-02T14:13:17", "title": "Attention? Attention! | Lil'Log", "body_markdown": "[Updated on 2018-10-28: Add\nPointer Network\nand the\nlink\nto my implementation of Transformer.]\n[Updated on 2018-11-06: Add a\nlink\nto the implementation of Transformer model.]\n[Updated on 2018-11-18: Add\nNeural Turing Machines\n.]\n[Updated on 2019-07-18: Correct the mistake on using the term “self-attention” when introducing the\nshow-attention-tell\npaper; moved it to\nSelf-Attention\nsection.]\n[Updated on 2020-04-07: A follow-up post on improved Transformer models is\nhere\n.]\nAttention is, to some extent, motivated by how we pay visual attention to different regions of an image or correlate words in one sentence. Take the picture of a Shiba Inu in Fig. 1 as an example.\nA Shiba Inu in a men’s outfit. The credit of the original photo goes to Instagram\n@mensweardog\n.\nHuman visual attention allows us to focus on a certain region with “high resolution” (i.e. look at the pointy ear in the yellow box) while perceiving the surrounding image in “low resolution” (i.e. now how about the snowy background and the outfit?), and then adjust the focal point or do the inference accordingly. Given a small patch of an image, pixels in the rest provide clues what should be displayed there. We expect to see a pointy ear in the yellow box because we have seen a dog’s nose, another pointy ear on the right, and Shiba’s mystery eyes (stuff in the red boxes). However, the sweater and blanket at the bottom would not be as helpful as those doggy features.\nSimilarly, we can explain the relationship between words in one sentence or close context. When we see “eating”, we expect to encounter a food word very soon. The color term describes the food, but probably not so much with “eating” directly.\nOne word \"attends\" to other words in the same sentence differently.\nIn a nutshell, attention in deep learning can be broadly interpreted as a vector of importance weights: in order to predict or infer one element, such as a pixel in an image or a word in a sentence, we estimate using the attention vector how strongly it is correlated with (or “\nattends to\n” as you may have read in many papers) other elements and take the sum of their values weighted by the attention vector as the approximation of the target.\nWhat’s Wrong with Seq2Seq Model?\n#\nThe\nseq2seq\nmodel was born in the field of language modeling (\nSutskever, et al. 2014\n). Broadly speaking, it aims to transform an input sequence (source) to a new one (target) and both sequences can be of arbitrary lengths. Examples of transformation tasks include machine translation between multiple languages in either text or audio, question-answer dialog generation, or even parsing sentences into grammar trees.\nThe seq2seq model normally has an encoder-decoder architecture, composed of:\nAn\nencoder\nprocesses the input sequence and compresses the information into a context vector (also known as sentence embedding or “thought” vector) of a\nfixed length\n. This representation is expected to be a good summary of the meaning of the\nwhole\nsource sequence.\nA\ndecoder\nis initialized with the context vector to emit the transformed output. The early work only used the last state of the encoder network as the decoder initial state.\nBoth the encoder and decoder are recurrent neural networks, i.e. using\nLSTM or GRU\nunits.\nThe encoder-decoder model, translating the sentence \"she is eating a green apple\" to Chinese. The visualization of both encoder and decoder is unrolled in time.\nA critical and apparent disadvantage of this fixed-length context vector design is incapability of remembering long sentences. Often it has forgotten the first part once it completes processing the whole input. The attention mechanism was born (\nBahdanau et al., 2015\n) to resolve this problem.\nBorn for Translation\n#\nThe attention mechanism was born to help memorize long source sentences in neural machine translation (\nNMT\n). Rather than building a single context vector out of the encoder’s last hidden state, the secret sauce invented by attention is to create shortcuts between the context vector and the entire source input. The weights of these shortcut connections are customizable for each output element.\nWhile the context vector has access to the entire input sequence, we don’t need to worry about forgetting. The alignment between the source and target is learned and controlled by the context vector. Essentially the context vector consumes three pieces of information:\nencoder hidden states;\ndecoder hidden states;\nalignment between source and target.\nThe encoder-decoder model with additive attention mechanism in\nBahdanau et al., 2015\n.\nDefinition\n#\nNow let’s define the attention mechanism introduced in NMT in a scientific way. Say, we have a source sequence $\\mathbf{x}$ of length $n$ and try to output a target sequence $\\mathbf{y}$ of length $m$:\n$$\n\\begin{aligned}\n\\mathbf{x} &= [x_1, x_2, \\dots, x_n] \\\\\n\\mathbf{y} &= [y_1, y_2, \\dots, y_m]\n\\end{aligned}\n$$\n(Variables in bold indicate that they are vectors; same for everything else in this post.)\nThe encoder is a\nbidirectional RNN\n(or other recurrent network setting of your choice) with a forward hidden state $\\overrightarrow{\\boldsymbol{h}}_i$ and a backward one $\\overleftarrow{\\boldsymbol{h}}_i$. A simple concatenation of two represents the encoder state. The motivation is to include both the preceding and following words in the annotation of one word.\n$$\n\\boldsymbol{h}_i = [\\overrightarrow{\\boldsymbol{h}}_i^\\top; \\overleftarrow{\\boldsymbol{h}}_i^\\top]^\\top, i=1,\\dots,n\n$$\nThe decoder network has hidden state $\\boldsymbol{s}_t=f(\\boldsymbol{s}_{t-1}, y_{t-1}, \\mathbf{c}_t)$ for the output word at position t, $t=1,\\dots,m$, where the context vector $\\mathbf{c}_t$ is a sum of hidden states of the input sequence, weighted by alignment scores:\n$$\n\\begin{aligned}\n\\mathbf{c}_t &= \\sum_{i=1}^n \\alpha_{t,i} \\boldsymbol{h}_i & \\small{\\text{; Context vector for output }y_t}\\\\\n\\alpha_{t,i} &= \\text{align}(y_t, x_i) & \\small{\\text{; How well two words }y_t\\text{ and }x_i\\text{ are aligned.}}\\\\\n&= \\frac{\\exp(\\text{score}(\\boldsymbol{s}_{t-1}, \\boldsymbol{h}_i))}{\\sum_{i'=1}^n \\exp(\\text{score}(\\boldsymbol{s}_{t-1}, \\boldsymbol{h}_{i'}))} & \\small{\\text{; Softmax of some predefined alignment score.}}.\n\\end{aligned}\n$$\nThe alignment model assigns a score $\\alpha_{t,i}$ to the pair of input at position i and output at position t, $(y_t, x_i)$, based on how well they match. The set of $\\{\\alpha_{t, i}\\}$ are weights defining how much of each source hidden state should be considered for each output. In Bahdanau’s paper, the alignment score $\\alpha$ is parametrized by a\nfeed-forward network\nwith a single hidden layer and this network is jointly trained with other parts of the model. The score function is therefore in the following form, given that tanh is used as the non-linear activation function:\n$$\n\\text{score}(\\boldsymbol{s}_t, \\boldsymbol{h}_i) = \\mathbf{v}_a^\\top \\tanh(\\mathbf{W}_a[\\boldsymbol{s}_t; \\boldsymbol{h}_i])\n$$\nwhere both $\\mathbf{v}_a$ and $\\mathbf{W}_a$ are weight matrices to be learned in the alignment model.\nThe matrix of alignment scores is a nice byproduct to explicitly show the correlation between source and target words.\nAlignment matrix of \"L'accord sur l'Espace économique européen a été signé en août 1992\" (French) and its English translation \"The agreement on the European Economic Area was signed in August 1992\". (Image source: Fig 3 in\nBahdanau et al., 2015\n)\nCheck out this nice\ntutorial\nby Tensorflow team for more implementation instructions.\nA Family of Attention Mechanisms\n#\nWith the help of the attention, the dependencies between source and target sequences are not restricted by the in-between distance anymore! Given the big improvement by attention in machine translation, it soon got extended into the computer vision field (\nXu et al. 2015\n) and people started exploring various other forms of attention mechanisms (\nLuong, et al., 2015\n;\nBritz et al., 2017\n;\nVaswani, et al., 2017\n).\nSummary\n#\nBelow is a summary table of several popular attention mechanisms and corresponding alignment score functions:\nName\nAlignment score function\nCitation\nContent-base attention\n$\\text{score}(\\boldsymbol{s}_t, \\boldsymbol{h}_i) = \\text{cosine}[\\boldsymbol{s}_t, \\boldsymbol{h}_i]$\nGraves2014\nAdditive(*)\n$\\text{score}(\\boldsymbol{s}_t, \\boldsymbol{h}_i) = \\mathbf{v}_a^\\top \\tanh(\\mathbf{W}_a[\\boldsymbol{s}_{t-1}; \\boldsymbol{h}_i])$\nBahdanau2015\nLocation-Base\n$\\alpha_{t,i} = \\text{softmax}(\\mathbf{W}_a \\boldsymbol{s}_t)$\nNote: This simplifies the softmax alignment to only depend on the target position.\nLuong2015\nGeneral\n$\\text{score}(\\boldsymbol{s}_t, \\boldsymbol{h}_i) = \\boldsymbol{s}_t^\\top\\mathbf{W}_a\\boldsymbol{h}_i$\nwhere $\\mathbf{W}_a$ is a trainable weight matrix in the attention layer.\nLuong2015\nDot-Product\n$\\text{score}(\\boldsymbol{s}_t, \\boldsymbol{h}_i) = \\boldsymbol{s}_t^\\top\\boldsymbol{h}_i$\nLuong2015\nScaled Dot-Product(^)\n$\\text{score}(\\boldsymbol{s}_t, \\boldsymbol{h}_i) = \\frac{\\boldsymbol{s}_t^\\top\\boldsymbol{h}_i}{\\sqrt{n}}$\nNote: very similar to the dot-product attention except for a scaling factor; where n is the dimension of the source hidden state.\nVaswani2017\n(*) Referred to as “concat” in Luong, et al., 2015 and as “additive attention” in Vaswani, et al., 2017.\n(^) It adds a scaling factor $1/\\sqrt{n}$, motivated by the concern when the input is large, the softmax function may have an extremely small gradient, hard for efficient learning.\nHere are a summary of broader categories of attention mechanisms:\nName\nDefinition\nCitation\nSelf-Attention(&)\nRelating different positions of the same input sequence. Theoretically the self-attention can adopt any score functions above, but just replace the target sequence with the same input sequence.\nCheng2016\nGlobal/Soft\nAttending to the entire input state space.\nXu2015\nLocal/Hard\nAttending to the part of input state space; i.e. a patch of the input image.\nXu2015\n;\nLuong2015\n(&) Also, referred to as “intra-attention” in Cheng et al., 2016 and some other papers.\nSelf-Attention\n#\nSelf-attention\n, also known as\nintra-attention\n, is an attention mechanism relating different positions of a single sequence in order to compute a representation of the same sequence. It has been shown to be very useful in machine reading, abstractive summarization, or image description generation.\nThe\nlong short-term memory network\npaper used self-attention to do machine reading. In the example below, the self-attention mechanism enables us to learn the correlation between the current words and the previous part of the sentence.\nThe current word is in red and the size of the blue shade indicates the activation level. (Image source:\nCheng et al., 2016\n)\nSoft vs Hard Attention\n#\nIn the\nshow, attend and tell\npaper, attention mechanism is applied to images to generate captions. The image is first encoded by a CNN to extract features. Then a LSTM decoder consumes the convolution features to produce descriptive words one by one, where the weights are learned through attention. The visualization of the attention weights clearly demonstrates which regions of the image the model is paying attention to so as to output a certain word.\n\"A woman is throwing a frisbee in a park.\" (Image source: Fig. 6(b) in\nXu et al. 2015\n)\nThis paper first proposed the distinction between “soft” vs “hard” attention, based on whether the attention has access to the entire image or only a patch:\nSoft\nAttention: the alignment weights are learned and placed “softly” over all patches in the source image; essentially the same type of attention as in\nBahdanau et al., 2015\n.\nPro\n: the model is smooth and differentiable.\nCon\n: expensive when the source input is large.\nHard\nAttention: only selects one patch of the image to attend to at a time.\nPro\n: less calculation at the inference time.\nCon\n: the model is non-differentiable and requires more complicated techniques such as variance reduction or reinforcement learning to train. (\nLuong, et al., 2015\n)\nGlobal vs Local Attention\n#\nLuong, et al., 2015\nproposed the “global” and “local” attention. The global attention is similar to the soft attention, while the local one is an interesting blend between\nhard and soft\n, an improvement over the hard attention to make it differentiable: the model first predicts a single aligned position for the current target word and a window centered around the source position is then used to compute a context vector.\nGlobal vs local attention (Image source: Fig 2 & 3 in\nLuong, et al., 2015\n)\nNeural Turing Machines\n#\nAlan Turing in\n1936\nproposed a minimalistic model of computation. It is composed of a infinitely long tape and a head to interact with the tape. The tape has countless cells on it, each filled with a symbol: 0, 1 or blank (\" “). The operation head can read symbols, edit symbols and move left/right on the tape. Theoretically a Turing machine can simulate any computer algorithm, irrespective of how complex or expensive the procedure might be. The infinite memory gives a Turing machine an edge to be mathematically limitless. However, infinite memory is not feasible in real modern computers and then we only consider Turing machine as a mathematical model of computation.\nHow a Turing machine looks like: a tape + a head that handles the tape. (Image source:\nhttp://aturingmachine.com/\n)\nNeural Turing Machine\n(\nNTM\n,\nGraves, Wayne & Danihelka, 2014\n) is a model architecture for coupling a neural network with external memory storage. The memory mimics the Turing machine tape and the neural network controls the operation heads to read from or write to the tape. However, the memory in NTM is finite, and thus it probably looks more like a “Neural\nvon Neumann\nMachine”.\nNTM contains two major components, a\ncontroller\nneural network and a\nmemory\nbank.\nController: is in charge of executing operations on the memory. It can be any type of neural network, feed-forward or recurrent.\nMemory: stores processed information. It is a matrix of size $N \\times M$, containing N vector rows and each has $M$ dimensions.\nIn one update iteration, the controller processes the input and interacts with the memory bank accordingly to generate output. The interaction is handled by a set of parallel\nread\nand\nwrite\nheads. Both read and write operations are “blurry” by softly attending to all the memory addresses.\nFig 10. Neural Turing Machine Architecture.\nReading and Writing\n#\nWhen reading from the memory at time t, an attention vector of size $N$, $\\mathbf{w}_t$ controls how much attention to assign to different memory locations (matrix rows). The read vector $\\mathbf{r}_t$ is a sum weighted by attention intensity:\n$$\n\\mathbf{r}_t = \\sum_{i=1}^N w_t(i)\\mathbf{M}_t(i)\\text{, where }\\sum_{i=1}^N w_t(i)=1, \\forall i: 0 \\leq w_t(i) \\leq 1\n$$\nwhere $w_t(i)$ is the $i$-th element in $\\mathbf{w}_t$ and $\\mathbf{M}_t(i)$ is the $i$-th row vector in the memory.\nWhen writing into the memory at time t, as inspired by the input and forget gates in LSTM, a write head first wipes off some old content according to an erase vector $\\mathbf{e}_t$ and then adds new information by an add vector $\\mathbf{a}_t$.\n$$\n\\begin{aligned}\n\\tilde{\\mathbf{M}}_t(i) &= \\mathbf{M}_{t-1}(i) [\\mathbf{1} - w_t(i)\\mathbf{e}_t] &\\scriptstyle{\\text{; erase}}\\\\\n\\mathbf{M}_t(i) &= \\tilde{\\mathbf{M}}_t(i) + w_t(i) \\mathbf{a}_t &\\scriptstyle{\\text{; add}}\n\\end{aligned}\n$$\nAttention Mechanisms\n#\nIn Neural Turing Machine, how to generate the attention distribution $\\mathbf{w}_t$ depends on the addressing mechanisms: NTM uses a mixture of content-based and location-based addressings.\nContent-based addressing\nThe content-addressing creates attention vectors based on the similarity between the key vector $\\mathbf{k}_t$ extracted by the controller from the input and memory rows. The content-based attention scores are computed as cosine similarity and then normalized by softmax. In addition, NTM adds a strength multiplier $\\beta_t$ to amplify or attenuate the focus of the distribution.\n$$\nw_t^c(i) \n= \\text{softmax}(\\beta_t \\cdot \\text{cosine}[\\mathbf{k}_t, \\mathbf{M}_t(i)])\n= \\frac{\\exp(\\beta_t \\frac{\\mathbf{k}_t \\cdot \\mathbf{M}_t(i)}{\\|\\mathbf{k}_t\\| \\cdot \\|\\mathbf{M}_t(i)\\|})}{\\sum_{j=1}^N \\exp(\\beta_t \\frac{\\mathbf{k}_t \\cdot \\mathbf{M}_t(j)}{\\|\\mathbf{k}_t\\| \\cdot \\|\\mathbf{M}_t(j)\\|})}\n$$\nInterpolation\nThen an interpolation gate scalar $g_t$ is used to blend the newly generated content-based attention vector with the attention weights in the last time step:\n$$\n\\mathbf{w}_t^g = g_t \\mathbf{w}_t^c + (1 - g_t) \\mathbf{w}_{t-1} \n$$\nLocation-based addressing\nThe location-based addressing sums up the values at different positions in the attention vector, weighted by a weighting distribution over allowable integer shifts. It is equivalent to a 1-d convolution with a kernel $\\mathbf{s}_t(.)$, a function of the position offset. There are multiple ways to define this distribution. See for inspiration.\nTwo ways to represent the shift weighting distribution $\\mathbf{s}\\_t$.\nFinally the attention distribution is enhanced by a sharpening scalar $\\gamma_t \\geq 1$.\n$$\n\\begin{aligned}\n\\tilde{w}_t(i) &= \\sum_{j=1}^N w_t^g(j) s_t(i-j) & \\scriptstyle{\\text{; circular convolution}}\\\\\nw_t(i) &= \\frac{\\tilde{w}_t(i)^{\\gamma_t}}{\\sum_{j=1}^N \\tilde{w}_t(j)^{\\gamma_t}} & \\scriptstyle{\\text{; sharpen}}\n\\end{aligned}\n$$\nThe complete process of generating the attention vector $\\mathbf{w}_t$ at time step t is illustrated in All the parameters produced by the controller are unique for each head. If there are multiple read and write heads in parallel, the controller would output multiple sets.\nFlow diagram of the addressing mechanisms in Neural Turing Machine. (Image source:\nGraves, Wayne & Danihelka, 2014\n)\nPointer Network\n#\nIn problems like sorting or travelling salesman, both input and output are sequential data. Unfortunately, they cannot be easily solved by classic seq-2-seq or NMT models, given that the discrete categories of output elements are not determined in advance, but depends on the variable input size. The\nPointer Net\n(\nPtr-Net\n;\nVinyals, et al. 2015\n) is proposed to resolve this type of problems: When the output elements correspond to\npositions\nin an input sequence. Rather than using attention to blend hidden units of an encoder into a context vector (See Fig. 8), the Pointer Net applies attention over the input elements to pick one as the output at each decoder step.\nThe architecture of a Pointer Network model. (Image source:\nVinyals, et al. 2015\n)\nThe Ptr-Net outputs a sequence of integer indices, $\\boldsymbol{c} = (c_1, \\dots, c_m)$ given a sequence of input vectors $\\boldsymbol{x} = (x_1, \\dots, x_n)$ and $1 \\leq c_i \\leq n$. The model still embraces an encoder-decoder framework. The encoder and decoder hidden states are denoted as $(\\boldsymbol{h}_1, \\dots, \\boldsymbol{h}_n)$ and $(\\boldsymbol{s}_1, \\dots, \\boldsymbol{s}_m)$, respectively. Note that $\\mathbf{s}_i$ is the output gate after cell activation in the decoder. The Ptr-Net applies additive attention between states and then normalizes it by softmax to model the output conditional probability:\n$$\n\\begin{aligned}\ny_i &= p(c_i \\vert c_1, \\dots, c_{i-1}, \\boldsymbol{x}) \\\\\n &= \\text{softmax}(\\text{score}(\\boldsymbol{s}_t; \\boldsymbol{h}_i)) = \\text{softmax}(\\mathbf{v}_a^\\top \\tanh(\\mathbf{W}_a[\\boldsymbol{s}_t; \\boldsymbol{h}_i]))\n\\end{aligned}\n$$\nThe attention mechanism is simplified, as Ptr-Net does not blend the encoder states into the output with attention weights. In this way, the output only responds to the positions but not the input content.\nTransformer\n#\n“Attention is All you Need”\n(Vaswani, et al., 2017), without a doubt, is one of the most impactful and interesting paper in 2017. It presented a lot of improvements to the soft attention and make it possible to do seq2seq modeling\nwithout\nrecurrent network units. The proposed “\ntransformer\n” model is entirely built on the self-attention mechanisms without using sequence-aligned recurrent architecture.\nThe secret recipe is carried in its model architecture.\nKey, Value and Query\n#\nThe major component in the transformer is the unit of\nmulti-head self-attention mechanism\n. The transformer views the encoded representation of the input as a set of\nkey\n-\nvalue\npairs, $(\\mathbf{K}, \\mathbf{V})$, both of dimension $n$ (input sequence length); in the context of NMT, both the keys and values are the encoder hidden states. In the decoder, the previous output is compressed into a\nquery\n($\\mathbf{Q}$ of dimension $m$) and the next output is produced by mapping this query and the set of keys and values.\nThe transformer adopts the\nscaled dot-product attention\n: the output is a weighted sum of the values, where the weight assigned to each value is determined by the dot-product of the query with all the keys:\n$$\n\\text{Attention}(\\mathbf{Q}, \\mathbf{K}, \\mathbf{V}) = \\text{softmax}(\\frac{\\mathbf{Q}\\mathbf{K}^\\top}{\\sqrt{n}})\\mathbf{V}\n$$\nMulti-Head Self-Attention\n#\nMulti-head scaled dot-product attention mechanism. (Image source: Fig 2 in\nVaswani, et al., 2017\n)\nRather than only computing the attention once, the multi-head mechanism runs through the scaled dot-product attention multiple times in parallel. The independent attention outputs are simply concatenated and linearly transformed into the expected dimensions. I assume the motivation is because ensembling always helps? ;) According to the paper,\n“multi-head attention allows the model to jointly attend to information from different representation\nsubspaces\nat different positions. With a single attention head, averaging inhibits this.”\n$$\n\\begin{aligned}\n\\text{MultiHead}(\\mathbf{Q}, \\mathbf{K}, \\mathbf{V}) &= [\\text{head}_1; \\dots; \\text{head}_h]\\mathbf{W}^O \\\\\n\\text{where head}_i &= \\text{Attention}(\\mathbf{Q}\\mathbf{W}^Q_i, \\mathbf{K}\\mathbf{W}^K_i, \\mathbf{V}\\mathbf{W}^V_i)\n\\end{aligned}\n$$\nwhere $\\mathbf{W}^Q_i$, $\\mathbf{W}^K_i$, $\\mathbf{W}^V_i$, and $\\mathbf{W}^O$ are parameter matrices to be learned.\nEncoder\n#\nThe transformer’s encoder. (Image source:\nVaswani, et al., 2017\n)\nThe encoder generates an attention-based representation with capability to locate a specific piece of information from a potentially infinitely-large context.\nA stack of N=6 identical layers.\nEach layer has a\nmulti-head self-attention layer\nand a simple position-wise\nfully connected feed-forward network\n.\nEach sub-layer adopts a\nresidual\nconnection and a layer\nnormalization\n.\nAll the sub-layers output data of the same dimension $d_\\text{model} = 512$.\nDecoder\n#\nThe transformer’s decoder. (Image source:\nVaswani, et al., 2017\n)\nThe decoder is able to retrieval from the encoded representation.\nA stack of N = 6 identical layers\nEach layer has two sub-layers of multi-head attention mechanisms and one sub-layer of fully-connected feed-forward network.\nSimilar to the encoder, each sub-layer adopts a residual connection and a layer normalization.\nThe first multi-head attention sub-layer is\nmodified\nto prevent positions from attending to subsequent positions, as we don’t want to look into the future of the target sequence when predicting the current position.\nFull Architecture\n#\nFinally here is the complete view of the transformer’s architecture:\nBoth the source and target sequences first go through embedding layers to produce data of the same dimension $d_\\text{model} =512$.\nTo preserve the position information, a sinusoid-wave-based positional encoding is applied and summed with the embedding output.\nA softmax and linear layer are added to the final decoder output.\nThe full model architecture of the transformer. (Image source: Fig 1 & 2 in\nVaswani, et al., 2017\n.)\nTry to implement the transformer model is an interesting experience, here is mine:\nlilianweng/transformer-tensorflow\n. Read the comments in the code if you are interested.\nSNAIL\n#\nThe transformer has no recurrent or convolutional structure, even with the positional encoding added to the embedding vector, the sequential order is only weakly incorporated. For problems sensitive to the positional dependency like\nreinforcement learning\n, this can be a big problem.\nThe\nSimple Neural Attention\nMeta-Learner\n(\nSNAIL\n) (\nMishra et al., 2017\n) was developed partially to resolve the problem with\npositioning\nin the transformer model by combining the self-attention mechanism in transformer with\ntemporal convolutions\n. It has been demonstrated to be good at both supervised learning and reinforcement learning tasks.\nSNAIL model architecture (Image source:\nMishra et al., 2017\n)\nSNAIL was born in the field of meta-learning, which is another big topic worthy of a post by itself. But in simple words, the meta-learning model is expected to be generalizable to novel, unseen tasks in the similar distribution. Read\nthis\nnice introduction if interested.\nSelf-Attention GAN\n#\nSelf-Attention GAN\n(\nSAGAN\n;\nZhang et al., 2018\n) adds self-attention layers into\nGAN\nto enable both the generator and the discriminator to better model relationships between spatial regions.\nThe classic\nDCGAN\n(Deep Convolutional GAN) represents both discriminator and generator as multi-layer convolutional networks. However, the representation capacity of the network is restrained by the filter size, as the feature of one pixel is limited to a small local region. In order to connect regions far apart, the features have to be dilute through layers of convolutional operations and the dependencies are not guaranteed to be maintained.\nAs the (soft) self-attention in the vision context is designed to explicitly learn the relationship between one pixel and all other positions, even regions far apart, it can easily capture global dependencies. Hence GAN equipped with self-attention is expected to\nhandle details better\n, hooray!\nConvolution operation and self-attention have access to regions of very different sizes.\nThe SAGAN adopts the\nnon-local neural network\nto apply the attention computation. The convolutional image feature maps $\\mathbf{x}$ is branched out into three copies, corresponding to the concepts of\nkey, value, and query\nin the transformer:\nKey: $f(\\mathbf{x}) = \\mathbf{W}_f \\mathbf{x}$\nQuery: $g(\\mathbf{x}) = \\mathbf{W}_g \\mathbf{x}$\nValue: $h(\\mathbf{x}) = \\mathbf{W}_h \\mathbf{x}$\nThen we apply the dot-product attention to output the self-attention feature maps:\n$$\n\\begin{aligned}\n\\alpha_{i,j} &= \\text{softmax}(f(\\mathbf{x}_i)^\\top g(\\mathbf{x}_j)) \\\\\n\\mathbf{o}_j &= \\mathbf{W}_v \\Big( \\sum_{i=1}^N \\alpha_{i,j} h(\\mathbf{x}_i) \\Big)\n\\end{aligned}\n$$\nThe self-attention mechanism in SAGAN. (Image source: Fig. 2 in\nZhang et al., 2018\n)\nNote that $\\alpha_{i,j}$ is one entry in the attention map, indicating how much attention the model should pay to the $i$-th position when synthesizing the $j$-th location. $\\mathbf{W}_f$, $\\mathbf{W}_g$, and $\\mathbf{W}_h$ are all 1x1 convolution filters. If you feel that 1x1 conv sounds like a weird concept (i.e., isn’t it just to multiply the whole feature map with one number?), watch this short\ntutorial\nby Andrew Ng. The output $\\mathbf{o}_j$ is a column vector of the final output $\\mathbf{o}= (\\mathbf{o}_1, \\mathbf{o}_2, \\dots, \\mathbf{o}_j, \\dots, \\mathbf{o}_N)$.\nFurthermore, the output of the attention layer is multiplied by a scale parameter and added back to the original input feature map:\n$$\n\\mathbf{y} = \\mathbf{x}_i + \\gamma \\mathbf{o}_i\n$$\nWhile the scaling parameter $\\gamma$ is increased gradually from 0 during the training, the network is configured to first rely on the cues in the local regions and then gradually learn to assign more weight to the regions that are further away.\n128×128 example images generated by SAGAN for different classes. (Image source: Partial Fig. 6 in\nZhang et al., 2018\n)\nCited as:\n@article{weng2018attention,\n title = \"Attention? Attention!\",\n author = \"Weng, Lilian\",\n journal = \"lilianweng.github.io\",\n year = \"2018\",\n url = \"https://lilianweng.github.io/posts/2018-06-24-attention/\"\n}\nReferences\n#\n[1]\n“Attention and Memory in Deep Learning and NLP.”\n- Jan 3, 2016 by Denny Britz\n[2]\n“Neural Machine Translation (seq2seq) Tutorial”\n[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.\n“Neural machine translation by jointly learning to align and translate.”\nICLR 2015.\n[4] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.\n“Show, attend and tell: Neural image caption generation with visual attention.”\nICML, 2015.\n[5] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.\n“Sequence to sequence learning with neural networks.”\nNIPS 2014.\n[6] Thang Luong, Hieu Pham, Christopher D. Manning.\n“Effective Approaches to Attention-based Neural Machine Translation.”\nEMNLP 2015.\n[7] Denny Britz, Anna Goldie, Thang Luong, and Quoc Le.\n“Massive exploration of neural machine translation architectures.”\nACL 2017.\n[8] Ashish Vaswani, et al.\n“Attention is all you need.”\nNIPS 2017.\n[9] Jianpeng Cheng, Li Dong, and Mirella Lapata.\n“Long short-term memory-networks for machine reading.”\nEMNLP 2016.\n[10] Xiaolong Wang, et al.\n“Non-local Neural Networks.”\nCVPR 2018\n[11] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.\n“Self-Attention Generative Adversarial Networks.”\narXiv preprint arXiv:1805.08318 (2018).\n[12] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel.\n“A simple neural attentive meta-learner.”\nICLR 2018.\n[13]\n“WaveNet: A Generative Model for Raw Audio”\n- Sep 8, 2016 by DeepMind.\n[14] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly.\n“Pointer networks.”\nNIPS 2015.\n[15] Alex Graves, Greg Wayne, and Ivo Danihelka.\n“Neural turing machines.”\narXiv preprint arXiv:1410.5401 (2014).", "metadata": { "title": "Attention? Attention! | Lil'Log", "description": "[Updated on 2018-10-28: Add Pointer Network and the link to my implementation of Transformer.]\n[Updated on 2018-11-06: Add a link to the implementation of Transformer model.]\n[Updated on 2018-11-18: Add Neural Turing Machines.]\n[Updated on 2019-07-18: Correct the mistake on using the term “self-attention” when introducing the show-attention-tell paper; moved it to Self-Attention section.]\n[Updated on 2020-04-07: A follow-up post on improved Transformer models is here.]", "keywords": "architecture, attention, transformer, rnn", "author": "Lilian Weng", "og:title": "Attention? Attention!", "og:description": "[Updated on 2018-10-28: Add Pointer Network and the link to my implementation of Transformer.]\n[Updated on 2018-11-06: Add a link to the implementation of Transformer model.]\n[Updated on 2018-11-18: Add Neural Turing Machines.]\n[Updated on 2019-07-18: Correct the mistake on using the term “self-attention” when introducing the show-attention-tell paper; moved it to Self-Attention section.]\n[Updated on 2020-04-07: A follow-up post on improved Transformer models is here.]", "og:type": "article", "og:url": "https://lilianweng.github.io/posts/2018-06-24-attention/", "twitter:card": "summary", "twitter:title": "Attention? Attention!", "twitter:description": "[Updated on 2018-10-28: Add Pointer Network and the link to my implementation of Transformer.]\n[Updated on 2018-11-06: Add a link to the implementation of Transformer model.]\n[Updated on 2018-11-18: Add Neural Turing Machines.]\n[Updated on 2019-07-18: Correct the mistake on using the term “self-attention” when introducing the show-attention-tell paper; moved it to Self-Attention section.]\n[Updated on 2020-04-07: A follow-up post on improved Transformer models is here.]", "article:section": "posts", "article:published_time": "2018-06-24T00:00:00+00:00", "article:modified_time": "2018-06-24T00:00:00+00:00" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/1c9291c74af3/code/body-000.txt", "chars": 222, "language": "text" } ], "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/shiba-example-attention.png", "bytes": 2159899, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-8571ab535bf08b3eb897.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/shiba-example-attention.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/sentence-example-attention.png", "bytes": 36779, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-244663c064863ba3add1.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/sentence-example-attention.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/encoder-decoder-example.png", "bytes": 163006, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-417a080a721a8da35e76.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/encoder-decoder-example.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/encoder-decoder-attention.png", "bytes": 360658, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-7a8e772d26ca2075d030.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/encoder-decoder-attention.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/bahdanau-fig3.png", "bytes": 329846, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-a79121328d38f1e7decb.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/bahdanau-fig3.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/cheng2016-fig1.png", "bytes": 1061108, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-217a917d5b695d69920e.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/cheng2016-fig1.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/xu2015-fig6b.png", "bytes": 3290656, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-16ff0e0557cd1ec7183b.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/xu2015-fig6b.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/luong2015-fig2-3.png", "bytes": 190907, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-68ed274d29e23ec04058.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/luong2015-fig2-3.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/turing-machine.jpg", "bytes": 20601, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-f7ff47f8ed0c85c31c04.jpg", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/turing-machine.jpg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/NTM.png", "bytes": 194344, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-d885b9f00fc903916eff.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/NTM.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/shift-weighting.png", "bytes": 236994, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-43b2947b31f475b9d117.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/shift-weighting.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/NTM-flow-addressing.png", "bytes": 476899, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-202d6793b7b96d2f655a.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/NTM-flow-addressing.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/ptr-net.png", "bytes": 401199, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-47a02b382f80dc9a2137.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/ptr-net.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/multi-head-attention.png", "bytes": 106503, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-d314a0ab86b5abc9e45b.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/multi-head-attention.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/transformer-encoder.png", "bytes": 136297, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-5bee749c4008d551aef3.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/transformer-encoder.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/transformer-decoder.png", "bytes": 198560, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-9686a8baee36d438f09c.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/transformer-decoder.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/transformer.png", "bytes": 376109, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-7cfc826648a35a656e03.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/transformer.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/snail.png", "bytes": 485628, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-4f9fa106ff81362212ba.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/snail.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/conv-vs-self-attention.png", "bytes": 124592, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-3442cdcfc8a317d5e9cd.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/conv-vs-self-attention.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/SAGAN.png", "bytes": 392406, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-d18eeaedd090128a06bf.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/SAGAN.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2018-06-24-attention/SAGAN-examples.png", "bytes": 3409897, "reused": false, "local_path": "attachments/1c9291c74af3/images/body-82aaa856d170c7aa0cea.png", "src": "https://lilianweng.github.io/posts/2018-06-24-attention/SAGAN-examples.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://github.com/lilianweng/transformer-tensorflow", "label": "link", "type": "a" }, { "href": "https://arxiv.org/abs/1502.03044", "label": "show-attention-tell", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2020-04-07-the-transformer-family/", "label": "here", "type": "a" }, { "href": "https://www.instagram.com/mensweardog/?hl=en", "label": "@mensweardog", "type": "a" }, { "href": "https://arxiv.org/abs/1409.3215", "label": "Sutskever, et al. 2014", "type": "a" }, { "href": "http://colah.github.io/posts/2015-08-Understanding-LSTMs/", "label": "LSTM or GRU", "type": "a" }, { "href": "https://arxiv.org/pdf/1409.0473.pdf", "label": "Bahdanau et al., 2015", "type": "a" }, { "href": "https://www.coursera.org/lecture/nlp-sequence-models/bidirectional-rnn-fyXnn", "label": "bidirectional RNN", "type": "a" }, { "href": "https://www.tensorflow.org/versions/master/tutorials/seq2seq", "label": "tutorial", "type": "a" }, { "href": "http://proceedings.mlr.press/v37/xuc15.pdf", "label": "Xu et al. 2015", "type": "a" }, { "href": "https://arxiv.org/pdf/1508.04025.pdf", "label": "Luong, et al., 2015", "type": "a" }, { "href": "https://arxiv.org/abs/1703.03906", "label": "Britz et al., 2017", "type": "a" }, { "href": "http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf", "label": "Vaswani, et al., 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1410.5401", "label": "Graves2014", "type": "a" }, { "href": "https://arxiv.org/pdf/1508.4025.pdf", "label": "Luong2015", "type": "a" }, { "href": "https://arxiv.org/pdf/1601.06733.pdf", "label": "Cheng2016", "type": "a" }, { "href": "https://arxiv.org/abs/1409.0473", "label": "Bahdanau et al., 2015", "type": "a" }, { "href": "https://arxiv.org/abs/1508.04025", "label": "Luong, et al., 2015", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Turing_machine", "label": "1936", "type": "a" }, { "href": "http://aturingmachine.com/", "label": "http://aturingmachine.com/", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Von_Neumann_architecture", "label": "von Neumann", "type": "a" }, { "href": "https://arxiv.org/abs/1506.03134", "label": "Vinyals, et al. 2015", "type": "a" }, { "href": "https://arxiv.org/pdf/1512.03385.pdf", "label": "residual", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/", "label": "reinforcement learning", "type": "a" }, { "href": "http://bair.berkeley.edu/blog/2017/07/18/learning-to-learn/", "label": "Meta-Learner", "type": "a" }, { "href": "http://metalearning.ml/papers/metalearn17_mishra.pdf", "label": "Mishra et al., 2017", "type": "a" }, { "href": "https://deepmind.com/blog/wavenet-generative-model-raw-audio/", "label": "temporal convolutions", "type": "a" }, { "href": "https://arxiv.org/pdf/1805.08318.pdf", "label": "Zhang et al., 2018", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2017-08-20-gan/", "label": "GAN", "type": "a" }, { "href": "https://arxiv.org/abs/1511.06434", "label": "DCGAN", "type": "a" }, { "href": "https://arxiv.org/pdf/1711.07971.pdf", "label": "non-local neural network", "type": "a" }, { "href": "https://arxiv.org/abs/1805.08318", "label": "Zhang et al., 2018", "type": "a" }, { "href": "https://www.coursera.org/lecture/convolutional-neural-networks/networks-in-networks-and-1x1-convolutions-ZTb8x", "label": "tutorial", "type": "a" }, { "href": "http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/", "label": "“Attention and Memory in Deep Learning and NLP.”", "type": "a" }, { "href": "https://github.com/tensorflow/nmt", "label": "“Neural Machine Translation (seq2seq) Tutorial”", "type": "a" }, { "href": "https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf", "label": "“Sequence to sequence learning with neural networks.”", "type": "a" }, { "href": "https://arxiv.org/abs/1707.03141", "label": "“A simple neural attentive meta-learner.”", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:51:32Z", "source_url": "https://lilianweng.github.io/posts/2018-06-24-attention/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/1502.03044", "bytes": 9582640, "reused": false, "local_path": "papers/repair/cc9ae4c67126750bbe41.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1409.3215", "bytes": 112084, "reused": false, "local_path": "papers/repair/61d552395d0f0789263b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1409.0473.pdf", "bytes": 444482, "reused": false, "local_path": "papers/repair/ffb5ff16a013792114ac.pdf" }, { "status": "ok", "url": "http://proceedings.mlr.press/v37/xuc15.pdf", "bytes": 3126761, "reused": false, "local_path": "papers/repair/53e1470cf7704d6e3cb9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1508.04025.pdf", "bytes": 249825, "reused": false, "local_path": "papers/repair/dac4f1215a64e5bf890e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1703.03906", "bytes": 305967, "reused": false, "local_path": "papers/repair/2864d7bf295692b04b3f.pdf" }, { "status": "ok", "url": "http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf", "bytes": 569417, "reused": true, "local_path": "papers/repair/95fa654499d86d006948.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1410.5401", "bytes": 1357690, "reused": false, "local_path": "papers/repair/968a0ef3e0f98e3e8588.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1508.4025.pdf", "bytes": 249825, "reused": false, "local_path": "papers/repair/538f66bff21c49b4a2af.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1601.06733.pdf", "bytes": 2520829, "reused": false, "local_path": "papers/repair/eab5cb0dbe082766664e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1409.0473", "bytes": 444482, "reused": true, "local_path": "papers/repair/b6909588ff7b9ebf375f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1508.04025", "bytes": 249825, "reused": true, "local_path": "papers/repair/94d3d49016ad40753c67.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1506.03134", "bytes": 469544, "reused": false, "local_path": "papers/repair/88d23cac0369adc66dcd.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1512.03385.pdf", "bytes": 819383, "reused": false, "local_path": "papers/repair/984099b2816e5988dbb2.pdf" }, { "status": "http_404", "url": "http://metalearning.ml/papers/metalearn17_mishra.pdf", "retried_at": "2026-10-20T18:09:22Z" }, { "status": "ok", "url": "https://arxiv.org/pdf/1805.08318.pdf", "bytes": 6338711, "reused": false, "local_path": "papers/repair/09d96a257954f78287a4.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1511.06434", "bytes": 7457198, "reused": false, "local_path": "papers/repair/adcbe83e3cd16cada0ea.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1711.07971.pdf", "bytes": 2853537, "reused": false, "local_path": "papers/repair/684df287e5824edb084a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1805.08318", "bytes": 6338711, "reused": false, "local_path": "papers/repair/72f7e2aac2061b4b2acd.pdf" }, { "status": "ok", "url": "https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf", "bytes": 142950, "reused": false, "local_path": "papers/repair/3cc5c95987280b914097.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1707.03141", "bytes": 1026799, "reused": false, "local_path": "papers/repair/75189519ab64f046e1aa.pdf" } ], "actual_video_files": [], "body_chars": 29634, "expected_body_images": 21, "discovery_status": "ok", "delivery_status": "partial", "missing_reason": "http_404", "raw_html": "attachments/1c9291c74af3/pages/source.html", "local_html": "attachments/1c9291c74af3/pages/article.html", "body_base_path": "attachments/1c9291c74af3/pages/", "body_markdown_path": "attachments/1c9291c74af3/pages/article.md" }