{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Christopher Olah / colah's blog",
  "title_listed": "Attention and Augmented Recurrent Neural Networks",
  "keywords": "注意力,RNN,增强",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "http://distill.pub/2016/augmented-rnns/",
  "publish_date": "2016-09",
  "corpus_stage": "预训练/SFT",
  "notes": "注意力与增强RNN（Distill）",
  "id": "c809afc5d1f7",
  "collected_at": "2026-10-02T14:28:18",
  "title": "Attention and Augmented Recurrent Neural Networks",
  "body_markdown": "Distill\nAbout\nPrize\nSubmit\nAttention and Augmented Recurrent Neural Networks\nChris Olah\nGoogle Brain\nShan Carter\nGoogle Brain\nSept. 8\n2016\nCitation:\nOlah & Carter, 2016\nRecurrent neural networks are one of the staples of deep learning, allowing neural networks to work with sequences of data like text, audio and video. They can be used to boil a sequence down into a high-level understanding, to annotate sequences, and even to generate new sequences from scratch!\nrnn\nx0\ny0\nx1\ny1\nx2\ny2\nx3\ny3\nOne cell...\ncan be used over...\nand over...\nand over...\nx4\ny4\nagain.\nThe basic RNN design struggles with longer sequences, but a special variant—\n“long short-term memory” networks\n[1]\n—can even work with these. Such models have been found to be very powerful, achieving remarkable results in many tasks including translation, voice recognition, and image captioning. As a result, recurrent neural networks have become very widespread in the last few years.\nAs this has happened, we’ve seen a growing number of attempts to augment RNNs with new properties. Four directions stand out as particularly exciting:\nNeural Turing\nMachines\nhave external memory that they can read and write to.\nAttentional\nInterfaces\nallow RNNs to focus on parts of their input.\nAdaptive\nComputation Time\nallows for varying amounts of computation per step.\nNeural\nProgrammers\ncan call functions, building programs as they run.\nIndividually, these techniques are all potent extensions of RNNs, but the really striking thing is that they can be combined, and seem to just be points in a broader space. Further, they all rely on the same underlying trick—something called attention—to work.\nOur guess is that these “augmented RNNs” will have an important role to play in extending deep learning’s capabilities over the coming years.\nNeural Turing Machines\nNeural Turing Machines\n[2]\ncombine a RNN with an external memory bank. Since vectors are the natural language of neural networks, the memory is an array of vectors:\nrnn\nMemory is an array of vectors.\nNetwork A\nwrites and reads\nfrom this memory\neach step.\nx0\ny0\nx1\ny1\nx2\ny2\nx3\ny3\nBut how does reading and writing work? The challenge is that we want to make them differentiable. In particular, we want to make them differentiable with respect to the location we read from or write to, so that we can learn where to read and write. This is tricky because memory addresses seem to be fundamentally discrete. NTMs take a very clever solution to this: every step, they read and write everywhere, just to different extents.\nAs an example, let’s focus on reading. Instead of specifying a single location, the RNN outputs an “attention distribution” that describes how we spread out the amount we care about different memory positions. As such, the result of the read operation is a weighted sum.\nrnn\nattention\nmemory\nThe RNN gives an attention distribution\nwhich describe how we spread out the\namount we care about different memory\npositions.\nThe read result is a weighted sum.\nSimilarly, we write everywhere at once to different extents. Again, an attention distribution describes how much we write at every location. We do this by having the new value of a position in memory be a convex combination of the old memory content and the write value, with the position between the two decided by the attention weight.\nrnn\nattention\nold memory\nnew memory\nwrite value\nThe RNN gives an attention distribution,\ndescribing how much we should change\neach memory position towards the write\nvalue.\nInstead of writing to one location, we write\neverywhere, just to different extents.\nBut how do NTMs decide which positions in memory to focus their attention on? They actually use a combination of two different methods: content-based attention and location-based attention. Content-based attention allows NTMs to search through their memory and focus on places that match what they’re looking for, while location-based attention allows relative movement in memory, enabling the NTM to loop.\nrnn\nFirst, the controller gives a query\nvector and each memory entry is\nscored for similarity with the\nquery.\nThe scores are then converted\ninto a distribution using softmax.\nNext, we interpolate the\nattention from the previous\ntime step.\nWe convolve the attention with\na shift filter—this allows the\ncontroller to move its focus.\nFinally, we sharpen the\nattention distribution. This\nfinal attention distribution is\nfed to the read or write\noperation.\nThe RNN gives an attention distribution,\ndescribing how much we should change\neach memory position towards the write\nvalue.\nmemory\nBlue shows high similarity,\npink high dissimilarity.\ninterpolation amount\nshift filter\nRNN controller\nattention mechanism\nquery vector\nattention from previous step\nnew attention distribution\nThis capability to read and write allows NTMs to perform many simple algorithms, previously beyond neural networks. For example, they can learn to store a long sequence in memory, and then loop over it, repeating it back repeatedly. As they do this, we can watch where they read and write, to better understand what they’re doing:\nSee more experiments in\n[3]\n. This figure is based on the Repeat Copy experiment.\nThey can also learn to mimic a lookup table, or even learn to sort numbers (although they kind of cheat)! On the other hand, they still can’t do many basic things, like add or multiply numbers.\nSince the original NTM paper, there have been a number of exciting papers exploring similar directions. The Neural GPU\n[4]\novercomes the NTM’s inability to add and multiply numbers.\nZaremba & Sutskever\n[5]\ntrain NTMs using reinforcement learning instead of the differentiable read/writes used by the original. Neural Random Access Machines\n[6]\nwork based on pointers. Some papers have explored differentiable data structures, like stacks and queues\n[7, 8]\n. And memory networks\n[9, 10]\nare another approach to attacking similar problems.\nIn some objective sense, many of the tasks these models can perform—such as learning how to add numbers—aren’t that objectively hard. The traditional program synthesis community would eat them for lunch. But neural networks are capable of many other things, and models like the Neural Turing Machine seem to have knocked away a very profound limit on their abilities.\nCode\nThere are a number of open source implementations of these models. Open source implementations of the Neural Turing Machine include\nTaehoon Kim’s\n(TensorFlow),\nShawn Tan’s\n(Theano),\nFumin’s\n(Go),\nKai Sheng Tai’s\n(Torch), and\nSnip’s\n(Lasagne). Code for the Neural GPU publication was open sourced and put in the\nTensorFlow Models repository\n. Open source implementations of Memory Networks include\nFacebook’s\n(Torch/Matlab),\nYerevaNN’s\n(Theano), and\nTaehoon Kim’s\n(TensorFlow).\nAttentional Interfaces\nWhen I’m translating a sentence, I pay special attention to the word I’m presently translating. When I’m transcribing an audio recording, I listen carefully to the segment I’m actively writing down. And if you ask me to describe the room I’m sitting in, I’ll glance around at the objects I’m describing as I do so.\nNeural networks can achieve this same behavior using\nattention\n, focusing on part of a subset of the information they’re given. For example, an RNN can attend over the output of another RNN. At every time step, it focuses on different positions in the other RNN.\nWe’d like attention to be differentiable, so that we can learn where to focus. To do this, we use the same trick Neural Turing Machines use: we focus everywhere, just to different extents.\nrnn\nNetwork B focuses on different\ninformation from network A at\nevery step.\nThe attention distribution is usually generated with content-based attention. The attending RNN generates a query describing what it wants to focus on. Each item is dot-producted with the query to produce a score, describing how well it matches the query. The scores are fed into a softmax to create the attention distribution.\nrnn\nThe attending RNN generates a\nquery describing what it wants\nto focus on.\nEach item is dot producted with the\nquery to produce a score, describing\nhow well it matches the query. The\nscores are fed into a softmax to\ncreate the attention distribution.\nOne use of attention between RNNs is translation\n[11]\n. A traditional sequence-to-sequence model has to boil the entire input down into a single vector and then expands it back out. Attention avoids this by allowing the RNN processing the input to pass along information about each word it sees, and then for the RNN generating the output to focus on words as they become relevant.\n0.50, 0.02, 0.04, 0.00, 0.00, 0.00, 0.00, 0.00, 0.01, 0.01, 0.00, 0.00, 0.00, 0.05, 0.05\n0.14, 0.89, 0.07, 0.00, 0.00, 0.00, 0.00, 0.00, 0.01, 0.06, 0.00, 0.00, 0.00, 0.02, 0.00\n0.02, 0.00, 0.56, 0.00, 0.00, 0.00, 0.00, 0.01, 0.00, 0.00, 0.00, 0.00, 0.00, 0.02, 0.00\n0.00, 0.00, 0.01, 0.61, 0.02, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00\n0.00, 0.00, 0.02, 0.03, 0.01, 0.21, 0.97, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.01, 0.00\n0.00, 0.00, 0.02, 0.07, 0.25, 0.76, 0.02, 0.04, 0.00, 0.00, 0.00, 0.00, 0.00, 0.02, 0.00\n0.02, 0.04, 0.03, 0.27, 0.71, 0.02, 0.00, 0.02, 0.00, 0.00, 0.02, 0.00, 0.00, 0.05, 0.01\n0.02, 0.00, 0.07, 0.00, 0.00, 0.00, 0.00, 0.60, 0.40, 0.01, 0.04, 0.00, 0.00, 0.13, 0.01\n0.13, 0.02, 0.02, 0.00, 0.01, 0.00, 0.00, 0.23, 0.52, 0.91, 0.07, 0.00, 0.00, 0.10, 0.00\n0.04, 0.00, 0.01, 0.00, 0.00, 0.00, 0.00, 0.01, 0.03, 0.00, 0.67, 0.00, 0.00, 0.16, 0.02\n0.02, 0.00, 0.04, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.14, 0.96, 0.00, 0.02, 0.07\n0.02, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.02, 0.02, 1.00, 0.02, 0.03\n0.02, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.01, 0.00, 0.01, 0.00, 0.00, 0.32, 0.11\n0.04, 0.00, 0.08, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.01, 0.00, 0.00, 0.04, 0.67\noutput text\nnetwork B\nnetwork A\ninput text\nDiagram derived from Fig. 3 of\nBahdanau,\net al.\n2014\nThis kind of attention between RNNs has a number of other applications. It can be used in voice recognition\n[12]\n, allowing one RNN to process the audio and then have another RNN skim over it, focusing on relevant parts as it generates a transcript.\noutput text\nnetwork B\nnetwork A\ninput audio\n0.01,0.02,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.02,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.02,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.02,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.02,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.09,0.11,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.78,0.33,0.04,0.02,0.02,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.05,0.61,0.37,0.78,0.07,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.05,0.21,0.07,0.74,0.31,0.00,0.01,0.01,0.01,0.02,0.01,0.02,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.04,0.02,0.02,0.47,0.98,0.02,0.01,0.01,0.02,0.01,0.02,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.96,0.98,0.28,0.02,0.02,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.02,0.01,0.01,0.01,0.00,0.01,0.01,0.40,0.49,0.61,0.30,0.02,0.02,0.01,0.01,0.01,0.02,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.05,0.07,0.09,0.18,0.96,0.89,0.87,0.54,0.02,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.02,0.10,0.05,0.13,0.01,0.03,0.05,0.10,0.14,0.32,0.13,0.01,0.02,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.12,0.60,0.04,0.03,0.02,0.02,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.05,0.41,0.66,0.02,0.02,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.94,0.72,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.02,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.04,0.76,0.02,0.01,0.01,0.01,0.01,0.02,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.77,0.45,0.66,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.04,0.01,0.00,0.02,0.13,0.98,0.97,0.02,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.09,0.00,0.01,0.01,0.01,0.02,0.90,0.02,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.08,0.10,0.01,0.01,0.01,0.03,0.96,0.19,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.13,0.02,0.01,0.01,0.01,0.01,0.60,0.18\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.02,0.60\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.02\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.02,0.02,0.02,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\n0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.00,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01\nFigure derived from\nChan,\net al.\n2015\nOther uses of this kind of attention include parsing text\n[13]\n, where it allows the model to glance at words as it generates the parse tree, and for conversational modeling\n[14]\n, where it lets the model focus on previous parts of the conversation as it generates its response.\nAttention can also be used on the interface between a convolutional neural network and an RNN. This allows the RNN to look at different position of an image every step. One popular use of this kind of attention is for image captioning. First, a conv net processes the image, extracting high-level features. Then an RNN runs, generating a description of the image. As it generates each word in the description, the RNN focuses on the conv net’s interpretation of the relevant parts of the image. We can explicitly visualize this:\nFigure from\n[3]\nMore broadly, attentional interfaces can be used whenever one wants to interface with a neural network that has a repeating structure in its output.\nAttentional interfaces have been found to be an extremely general and powerful technique, and are becoming increasingly widespread.\nAdaptive Computation Time\nStandard RNNs do the same amount of computation for each time step. This seems unintuitive. Surely, one should think more when things are hard? It also limits RNNs to doing\nO(n)\noperations for a list of length\nn\n.\nAdaptive Computation Time\n[15]\nis a way for RNNs to do different amounts of computation each step. The big picture idea is simple: allow the RNN to do multiple steps of computation for each time step.\nIn order for the network to learn how many steps to do, we want the number of steps to be differentiable. We achieve this with the same trick we used before: instead of deciding to run for a discrete number of steps, we have an attention distribution over the number of steps to run. The output is a weighted combination of the outputs of each step.\nrnn\nFor every time step the RNN can\ndo multiple computation steps.\nThe output is a weighted combination\nof the computation step outputs.\nThe process is repeated for\neach time step.\nA special bit is set to denote\nthe first computation step.\nThere are a few more details, which were left out in the previous diagram. Here’s a complete diagram of a time step with three computation steps.\nrnn\nThat’s a bit complicated, so let’s work through it step by step. At a high-level, we’re still running the RNN and outputting a weighted combination of the states:\nrnn\nThe weight for each step is determined by a “halting neuron.” It’s a sigmoid neuron that looks at the RNN state and gives a halting weight, which we can think of as the probability that we should stop at that step.\nrnn\nhalting neuron\nWe have a total budget for the halting weights of 1, so we track that budget along the top. When it gets to less than epsilon, we stop.\nrnn\nWhen we stop, might have some left over halting budget because we stop when it gets to less than epsilon. What should we do with it? Technically, it’s being given to future steps but we don’t want to compute those, so we attribute it to the last step.\nrnn\nWhen training Adaptive Computation Time models, one adds a “ponder cost” term to the cost function. This penalizes the model for the amount of computation it uses. The bigger you make this term, the more it will trade-off performance for lowering compute time.\nAdaptive Computation Time is a very new idea, but we believe that it, along with similar ideas, will be very important.\nCode\nThe only open source implementation of Adaptive Computation Time at the moment seems to be\nMark Neumann’s\n(TensorFlow).\nNeural Programmer\nNeural nets are excellent at many tasks, but they also struggle to do some basic things like arithmetic, which are trivial in normal approaches to computing. It would be really nice to have a way to fuse neural nets with normal programming, and get the best of both worlds.\nThe neural programmer\n[16]\nis one approach to this. It learns to create programs in order to solve a task. In fact, it learns to generate such programs\nwithout needing examples of correct programs\n. It discovers how to produce programs as a means to the end of accomplishing some task.\nThe actual model in the paper answers questions about tables by generating SQL-like programs to query the table. However, there are a number of details here that make it a bit complicated, so let’s start by imagining a slightly simpler model, which is given an arithmetic expression and generates a program to evaluate it.\nThe generated program is a sequence of operations. Each operation is defined to operate on the output of past operations. So an operation might be something like “add the output of the operation 2 steps ago and the output of the operation 1 step ago.” It’s more like a Unix pipe than a program with variables being assigned to and read from.\nrnn\nThe program is generated one operation at a time by a controller RNN. At each step, the controller RNN outputs a probability distribution for what the next operation should be. For example, we might be pretty sure we want to perform addition at the first time step, then have a hard time deciding whether we should multiply or divide at the second step, and so on...\nrnn\nAt each step the\ncontroller RNN outputs a\nprobability distribution.\nThe resulting distribution over operations can now be evaluated. Instead of running a single operation at each step, we do the usual attention trick of running all of them and then average the outputs together, weighted by the probability we ran that operation.\nrnn\nAt each step the\ncontroller RNN outputs a\nprobability distribution.\nWe run all of the\noperations and average\nthe outputs together.\nAs long as we can define derivatives through the operations, the program’s output is differentiable with respect to the probabilities. We can then define a loss, and train the neural net to produce programs that give the correct answer. In this way, the Neural Programmer learns to produce programs without examples of good programs. The only supervision is the answer the program should produce.\nThat’s the core idea of Neural Programmer, but the version in the paper answers questions about tables, rather than arithmetic expressions. There are a few additional neat tricks:\nMultiple Types:\nMany of the operations in the Neural Programmer deal with types other than scalar numbers. Some operations output selections of table columns or selections of cells.\nOnly outputs of the same type get merged together.\nReferencing Inputs:\nThe neural programmer needs to answer questions like “How many cities have a population greater than 1,000,000?” given a table of cities with a population column. To facilitate this, some operations allow the network to reference constants in the question they’re answering, or the names of columns. This referencing happens by attention, in the style of pointer networks\n[17]\n.\nThe Neural Programmer isn’t the only approach to having neural networks generate programs. Another lovely approach is the Neural Programmer-Interpreter\n[18]\nwhich can accomplish a number of very interesting tasks, but requires supervision in the form of correct programs.\nWe think that this general space, of bridging the gap between more traditional programming and neural networks is extremely important. While the Neural Programmer is clearly not the final solution, we think there are a lot of important lessons to be learned from it.\nCode\nThe more recent version of Neural Programmer for\nquestion answering\nhas been open sourced by its authors and is available as a\nTensorFlow Model\n. There is also an implementation of the Neural Programmer-Interpreter by\nKen Morishita\n(Keras).\nThe Big Picture\nA human with a piece of paper is, in some sense, much smarter than a human without. A human with mathematical notation can solve problems they otherwise couldn’t. Access to computers makes us capable of incredible feats that would otherwise be far beyond us.\nIn general, it seems like a lot of interesting forms of intelligence are an interaction between the creative heuristic intuition of humans and some more crisp and careful media, like language or equations. Sometimes, the medium is something that physically exists, and stores information for us, prevents us from making mistakes, or does computational heavy lifting. In other cases, the medium is a model in our head that we manipulate. Either way, it seems deeply fundamental to intelligence.\nRecent results in machine learning have started to have this flavor, combining the intuition of neural networks with something else. One approach is what one might call “heuristic search.” For example, AlphaGo\n[19]\nhas a model of how Go works and explores how the game could play out guided by neural network intuition. Similarly, DeepMath\n[20]\nuses neural networks as intuition for manipulating mathematical expressions. The “augmented RNNs” we’ve talked about in this article are another approach, where we connect RNNs to engineered media, in order to extend their general capabilities.\nInteracting with media naturally involves making a sequence of taking an action, observing, and taking more actions. This creates a major challenge: how do we learn which actions to take? That sounds like a reinforcement learning problem and we could certainly take that approach. But the reinforcement learning literature is really attacking the hardest version of this problem, and its solutions are hard to use. The wonderful thing about attention is that it gives us an easier way out of this problem by partially taking all actions to varying extents. This works because we can design media—like the NTM memory—to allow fractional actions and to be differentiable. Reinforcement learning has us take a single path, and try to learn from that. Attention takes every direction at a fork, and then merges the paths back together.\nA major weaknesses of attention is that we have to take every “action” every step. This causes the computational cost to grow linearly as you do things like increase the amount of memory in a Neural Turing Machine. One thing you could imagine doing is having your attention be sparse, so that you only have to touch some memories. However, it’s still challenging because you may want to do things like have your attention depend on the content of the memory, and doing that naively forces you to look at each memory. We’ve seen some initial attempts to attack this problem, such as\n[21]\n, but it seems like there’s a lot more to be done. If we could really make such sub-linear time attention work, that would be very powerful!\nAugmented recurrent neural networks, and the underlying technique of attention, are incredibly exciting. We look forward to seeing what happens next!\nAcknowledgments\nThank you to Maithra Raghu, Dario Amodei, Cassandra Xia, Luke Vilnis, Anna Goldie, Jesse Engel, Dan Mané, Natasha Jaques, Emma Pierson and Ian Goodfellow for their feedback and encouragement. We’re also very grateful to our team,\nGoogle Brain\n, for being extremely supportive of our project.\nReferences\nUnderstanding LSTM Networks\n[link]\nOlah, C., 2015.\nNeural Turing Machines\n[PDF]\nGraves, A., Wayne, G. and Danihelka, I., 2014. CoRR, Vol abs/1410.5401.\nShow, attend and tell: Neural image caption generation with visual attention\nXu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R.S. and Bengio, Y., 2015. arXiv preprint arXiv:1502.03044, Vol 2(3), pp. 5. CoRR.\nNeural GPUs Learn Algorithms\n[PDF]\nKaiser, L. and Sutskever, I., 2015. CoRR, Vol abs/1511.08228.\nReinforcement Learning Neural Turing Machines\n[PDF]\nZaremba, W. and Sutskever, I., 2015. CoRR, Vol abs/1505.00521.\nNeural Random-Access Machines\n[PDF]\nKurach, K., Andrychowicz, M. and Sutskever, I., 2015. CoRR, Vol abs/1511.06392.\nLearning to Transduce with Unbounded Memory\n[PDF]\nGrefenstette, E., Hermann, K.M., Suleyman, M. and Blunsom, P., 2015. Advances in Neural Information Processing Systems 28, pp. 1828—1836. Curran Associates, Inc.\nInferring Algorithmic Patterns with Stack-Augmented Recurrent Nets\n[PDF]\nJoulin, A. and Mikolov, T., 2015. Advances in Neural Information Processing Systems 28, pp. 190—198. Curran Associates, Inc.\nMemory Networks\n[PDF]\nWeston, J., Chopra, S. and Bordes, A., 2014. CoRR, Vol abs/1410.3916.\nAsk Me Anything: Dynamic Memory Networks for Natural Language Processing\n[PDF]\nKumar, A., Irsoy, O., Su, J., Bradbury, J., English, R., Pierce, B., Ondruska, P., Gulrajani, I. and Socher, R., 2015. CoRR, Vol abs/1506.07285.\nNeural machine translation by jointly learning to align and translate\nBahdanau, D., Cho, K. and Bengio, Y., 2014. arXiv preprint arXiv:1409.0473.\nListen, Attend and Spell\n[PDF]\nChan, W., Jaitly, N., Le, Q.V. and Vinyals, O., 2015. CoRR, Vol abs/1508.01211.\nGrammar as a foreign language\nVinyals, O., Kaiser, L., Koo, T., Petrov, S., Sutskever, I. and Hinton, G., 2015. Advances in Neural Information Processing Systems, pp. 2773—2781.\nA Neural Conversational Model\n[PDF]\nVinyals, O. and Le, Q.V., 2015. CoRR, Vol abs/1506.05869.\nAdaptive Computation Time for Recurrent Neural Networks\n[PDF]\nGraves, A., 2016. CoRR, Vol abs/1603.08983.\nNeural Programmer: Inducing Latent Programs with Gradient Descent\n[PDF]\nNeelakantan, A., Le, Q.V. and Sutskever, I., 2015. CoRR, Vol abs/1511.04834.\nPointer networks\nVinyals, O., Fortunato, M. and Jaitly, N., 2015. Advances in Neural Information Processing Systems, pp. 2692—2700.\nNeural Programmer-Interpreters\n[PDF]\nReed, S.E. and Freitas, N.d., 2015. CoRR, Vol abs/1511.06279.\nMastering the game of Go with deep neural networks and tree search\n[link]\nSilver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Driessche, G.v.d., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T. and Hassabis, D., 2016. Nature, Vol 529(7587), pp. 484—489. Nature Publishing Group.\nDOI: 10.1038/nature16961\nDeepMath - Deep Sequence Models for Premise Selection\n[PDF]\nAlemi, A.A., Chollet, F., Irving, G., Szegedy, C. and Urban, J., 2016. CoRR, Vol abs/1606.04442.\nLearning Efficient Algorithms with Hierarchical Attentive Memory\n[PDF]\nAndrychowicz, M. and Kurach, K., 2016. CoRR, Vol abs/1602.03218.\nUpdates and Corrections\nView all changes\nto this article since it was first published. If you see a mistake or want to suggest a change, please\ncreate an issue on GitHub\n.\nCitations and Reuse\nDiagrams and text are licensed under Creative Commons Attribution\nCC-BY 2.0\n, unless noted otherwise, with the\nsource available on GitHub\n. The figures that have been reused from other sources don't fall under this license and can be recognized by a note in their caption: “Figure from …”.\nFor attribution in academic contexts, please cite this work as\nOlah & Carter, \"Attention and Augmented Recurrent Neural Networks\", Distill, 2016. http://doi.org/10.23915/distill.00001\nBibTeX citation\n@article{olah2016attention,\n  author = {Olah, Chris and Carter, Shan},\n  title = {Attention and Augmented Recurrent Neural Networks},\n  journal = {Distill},\n  year = {2016},\n  url = {http://distill.pub/2016/augmented-rnns},\n  doi = {10.23915/distill.00001}\n}\nDistill\nis dedicated to clear explanations of machine learning\nAbout\nSubmit\nPrize\nArchive\nRSS\nGitHub\nTwitter\nISSN 2476-0757\nUnderstanding LSTM Networks\n[link]\nC. Olah.  2015.\nNeural Turing Machines\n[PDF]\nA. Graves, G. Wayne, I. Danihelka.\nCoRR, Vol abs/1410.5401. 2014.\nShow, attend and tell: Neural image caption generation with visual attention\nK. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R.S. Zemel, Y. Bengio.\narXiv preprint arXiv:1502.03044, Vol 2(3), pp. 5. CoRR. 2015.\nNeural GPUs Learn Algorithms\n[PDF]\nL. Kaiser, I. Sutskever.\nCoRR, Vol abs/1511.08228. 2015.\nReinforcement Learning Neural Turing Machines\n[PDF]\nW. Zaremba, I. Sutskever.\nCoRR, Vol abs/1505.00521. 2015.\nNeural Random-Access Machines\n[PDF]\nK. Kurach, M. Andrychowicz, I. Sutskever.\nCoRR, Vol abs/1511.06392. 2015.\nLearning to Transduce with Unbounded Memory\n[PDF]\nE. Grefenstette, K.M. Hermann, M. Suleyman, P. Blunsom.\nAdvances in Neural Information Processing Systems 28, pp. 1828—1836. Curran Associates, Inc. 2015.\nInferring Algorithmic Patterns with Stack-Augmented Recurrent Nets\n[PDF]\nA. Joulin, T. Mikolov.\nAdvances in Neural Information Processing Systems 28, pp. 190—198. Curran Associates, Inc. 2015.\nMemory Networks\n[PDF]\nJ. Weston, S. Chopra, A. Bordes.\nCoRR, Vol abs/1410.3916. 2014.\nAsk Me Anything: Dynamic Memory Networks for Natural Language Processing\n[PDF]\nA. Kumar, O. Irsoy, J. Su, J. Bradbury, R. English, B. Pierce, P. Ondruska, I. Gulrajani, R. Socher.\nCoRR, Vol abs/1506.07285. 2015.\nNeural machine translation by jointly learning to align and translate\nD. Bahdanau, K. Cho, Y. Bengio.\narXiv preprint arXiv:1409.0473. 2014.\nListen, Attend and Spell\n[PDF]\nW. Chan, N. Jaitly, Q.V. Le, O. Vinyals.\nCoRR, Vol abs/1508.01211. 2015.\nGrammar as a foreign language\nO. Vinyals, L. Kaiser, T. Koo, S. Petrov, I. Sutskever, G. Hinton.\nAdvances in Neural Information Processing Systems, pp. 2773—2781. 2015.\nA Neural Conversational Model\n[PDF]\nO. Vinyals, Q.V. Le.\nCoRR, Vol abs/1506.05869. 2015.\nShow, attend and tell: Neural image caption generation with visual attention\nK. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R.S. Zemel, Y. Bengio.\narXiv preprint arXiv:1502.03044, Vol 2(3), pp. 5. CoRR. 2015.\nAdaptive Computation Time for Recurrent Neural Networks\n[PDF]\nA. Graves.\nCoRR, Vol abs/1603.08983. 2016.\nNeural Programmer: Inducing Latent Programs with Gradient Descent\n[PDF]\nA. Neelakantan, Q.V. Le, I. Sutskever.\nCoRR, Vol abs/1511.04834. 2015.\nPointer networks\nO. Vinyals, M. Fortunato, N. Jaitly.\nAdvances in Neural Information Processing Systems, pp. 2692—2700. 2015.\nNeural Programmer-Interpreters\n[PDF]\nS.E. Reed, N.d. Freitas.\nCoRR, Vol abs/1511.06279. 2015.\nMastering the game of Go with deep neural networks and tree search\n[link]\nD. Silver, A. Huang, C.J. Maddison, A. Guez, L. Sifre, G.v.d. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, D. Hassabis.\nNature, Vol 529(7587), pp. 484—489. Nature Publishing Group. 2016.\nDOI: 10.1038/nature16961\nDeepMath - Deep Sequence Models for Premise Selection\n[PDF]\nA.A. Alemi, F. Chollet, G. Irving, C. Szegedy, J. Urban.\nCoRR, Vol abs/1606.04442. 2016.\nLearning Efficient Algorithms with Hierarchical Attentive Memory\n[PDF]\nM. Andrychowicz, K. Kurach.\nCoRR, Vol abs/1602.03218. 2016.",
  "metadata": {
    "title": "Attention and Augmented Recurrent Neural Networks",
    "description": null,
    "keywords": null,
    "author": null,
    "og:type": "article",
    "og:title": "Attention and Augmented Recurrent Neural Networks",
    "og:description": "A visual overview of neural attention, and the powerful extensions of neural networks being built on top of it.",
    "og:url": "http://distill.pub/2016/augmented-rnns",
    "og:image": "http://distill.pub/2016/augmented-rnns/thumbnail.jpg",
    "og:locale": "en_US",
    "og:site_name": "Distill",
    "twitter:card": "summary_large_image",
    "twitter:title": "Attention and Augmented Recurrent Neural Networks",
    "twitter:description": "A visual overview of neural attention, and the powerful extensions of neural networks being built on top of it.",
    "twitter:url": "http://distill.pub/2016/augmented-rnns",
    "twitter:image": "http://distill.pub/2016/augmented-rnns/thumbnail.jpg",
    "twitter:image:width": "560",
    "twitter:image:height": "295",
    "article:published": "2016-09-08",
    "article:created": "Thu Sep 08 2016 16:00:00 GMT-0400 (Eastern Daylight Time)",
    "article:modified": "Tue May 22 2018 14:19:28 GMT-0400 (Eastern Daylight Time)",
    "article:author": "Shan Carter"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/c809afc5d1f7/code/body-000.txt",
      "chars": 1245,
      "language": "text"
    },
    {
      "file": "attachments/c809afc5d1f7/code/body-001.txt",
      "chars": 6270,
      "language": "text"
    },
    {
      "file": "attachments/c809afc5d1f7/code/body-002.txt",
      "chars": 120,
      "language": "text"
    },
    {
      "file": "attachments/c809afc5d1f7/code/body-003.txt",
      "chars": 258,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://distill.pub/2016/augmented-rnns/assets/rnn_preview_ntm.svg",
      "bytes": 4574,
      "reused": false,
      "local_path": "attachments/c809afc5d1f7/images/body-ee9f6aa1fd7be8ade3df.svg",
      "src": "https://distill.pub/2016/augmented-rnns/assets/rnn_preview_ntm.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2016/augmented-rnns/assets/rnn_preview_ai.svg",
      "bytes": 4125,
      "reused": false,
      "local_path": "attachments/c809afc5d1f7/images/body-8777d14a16da9f2c848b.svg",
      "src": "https://distill.pub/2016/augmented-rnns/assets/rnn_preview_ai.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2016/augmented-rnns/assets/rnn_preview_act.svg",
      "bytes": 3581,
      "reused": false,
      "local_path": "attachments/c809afc5d1f7/images/body-0f2411fe6e29723faf5e.svg",
      "src": "https://distill.pub/2016/augmented-rnns/assets/rnn_preview_act.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2016/augmented-rnns/assets/rnn_preview_np.svg",
      "bytes": 12721,
      "reused": false,
      "local_path": "attachments/c809afc5d1f7/images/body-d87f44fe666f0e2f762f.svg",
      "src": "https://distill.pub/2016/augmented-rnns/assets/rnn_preview_np.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2016/augmented-rnns/assets/NTM-Copy-ReadWrite.svg",
      "bytes": 35440,
      "reused": false,
      "local_path": "attachments/c809afc5d1f7/images/body-516eb5bd2b2cc0b79c60.svg",
      "src": "https://distill.pub/2016/augmented-rnns/assets/NTM-Copy-ReadWrite.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2016/augmented-rnns/assets/pointer.svg",
      "bytes": 2920,
      "reused": false,
      "local_path": "attachments/c809afc5d1f7/images/body-261f1c29fb2db28ef95c.svg",
      "src": "https://distill.pub/2016/augmented-rnns/assets/pointer.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2016/augmented-rnns/assets/rnn-attention-waveform-bw.png",
      "bytes": 58956,
      "reused": false,
      "local_path": "attachments/c809afc5d1f7/images/body-e6028bd71419eb3f8a9b.png",
      "src": "https://distill.pub/2016/augmented-rnns/assets/rnn-attention-waveform-bw.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2016/augmented-rnns/assets/show-attend-tell.png",
      "bytes": 440849,
      "reused": false,
      "local_path": "attachments/c809afc5d1f7/images/body-5c561b6142add73485ea.png",
      "src": "https://distill.pub/2016/augmented-rnns/assets/show-attend-tell.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://distill.pub/",
      "label": "Distill",
      "type": "a"
    },
    {
      "href": "https://distill.pub/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "https://distill.pub/prize/",
      "label": "Prize",
      "type": "a"
    },
    {
      "href": "https://distill.pub/journal/",
      "label": "Submit",
      "type": "a"
    },
    {
      "href": "http://colah.github.io/",
      "label": "Chris Olah",
      "type": "a"
    },
    {
      "href": "http://g.co/brain",
      "label": "Google Brain",
      "type": "a"
    },
    {
      "href": "http://shancarter.com/",
      "label": "Shan Carter",
      "type": "a"
    },
    {
      "href": "http://colah.github.io/posts/2015-08-Understanding-LSTMs/",
      "label": "“long short-term memory” networks",
      "type": "a"
    },
    {
      "href": "https://github.com/carpedm20/NTM-tensorflow",
      "label": "Taehoon Kim’s",
      "type": "a"
    },
    {
      "href": "https://github.com/shawntan/neural-turing-machines",
      "label": "Shawn Tan’s",
      "type": "a"
    },
    {
      "href": "https://github.com/fumin/ntm",
      "label": "Fumin’s",
      "type": "a"
    },
    {
      "href": "https://github.com/kaishengtai/torch-ntm",
      "label": "Kai Sheng Tai’s",
      "type": "a"
    },
    {
      "href": "https://github.com/snipsco/ntm-lasagne",
      "label": "Snip’s",
      "type": "a"
    },
    {
      "href": "https://github.com/tensorflow/models/tree/master/neural_gpu",
      "label": "TensorFlow Models repository",
      "type": "a"
    },
    {
      "href": "https://github.com/facebook/MemNN",
      "label": "Facebook’s",
      "type": "a"
    },
    {
      "href": "https://github.com/YerevaNN/Dynamic-memory-networks-in-Theano",
      "label": "YerevaNN’s",
      "type": "a"
    },
    {
      "href": "https://github.com/carpedm20/MemN2N-tensorflow",
      "label": "Taehoon Kim’s",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1409.0473.pdf",
      "label": "Bahdanau, et al. 2014",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1508.01211.pdf",
      "label": "Chan, et al. 2015",
      "type": "a"
    },
    {
      "href": "https://github.com/DeNeutoy/act-tensorflow",
      "label": "Mark Neumann’s",
      "type": "a"
    },
    {
      "href": "https://openreview.net/pdf?id=ry2YOrcge",
      "label": "question answering",
      "type": "a"
    },
    {
      "href": "https://github.com/tensorflow/models/tree/master/neural_programmer",
      "label": "TensorFlow Model",
      "type": "a"
    },
    {
      "href": "https://github.com/mokemokechicken/keras_npi",
      "label": "Ken Morishita",
      "type": "a"
    },
    {
      "href": "http://colah.github.io/posts/2015-08-Understanding-LSTMs",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1410.5401.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1511.08228.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1505.00521.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1511.06392.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/5648-learning-to-transduce-with-unbounded-memory.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/5857-inferring-algorithmic-patterns-with-stack-augmented-recurrent-nets.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1410.3916.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1506.07285.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1508.01211.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1506.05869.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1603.08983.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1511.04834.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1511.06279.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://dx.doi.org/10.1038/nature16961",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.1038/nature16961",
      "label": "DOI: 10.1038/nature16961",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1606.04442.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1602.03218.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--augmented-rnns/compare/1596e094d8943d2dc0ea445d92071129c6419c59...fdc3526167133f5a97cee9e7cb48af293e67d66d",
      "label": "View all changes",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--augmented-rnns/issues/new",
      "label": "create an issue on GitHub",
      "type": "a"
    },
    {
      "href": "https://creativecommons.org/licenses/by/2.0/",
      "label": "CC-BY 2.0",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--augmented-rnns",
      "label": "source available on GitHub",
      "type": "a"
    },
    {
      "href": "http://distill.pub/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "http://distill.pub/journal/",
      "label": "Submit",
      "type": "a"
    },
    {
      "href": "http://distill.pub/prize/",
      "label": "Prize",
      "type": "a"
    },
    {
      "href": "http://distill.pub/archive/",
      "label": "Archive",
      "type": "a"
    },
    {
      "href": "http://distill.pub/rss.xml",
      "label": "RSS",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://twitter.com/distillpub",
      "label": "Twitter",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:34Z",
  "source_url": "http://distill.pub/2016/augmented-rnns/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1409.0473.pdf",
      "bytes": 444482,
      "reused": true,
      "local_path": "papers/repair/ffb5ff16a013792114ac.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1508.01211.pdf",
      "bytes": 2290560,
      "reused": false,
      "local_path": "papers/repair/4d108754dcd351549f8f.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1410.5401.pdf",
      "bytes": 1357690,
      "reused": false,
      "local_path": "papers/repair/95e4052b1259516ec37a.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1511.08228.pdf",
      "bytes": 139464,
      "reused": false,
      "local_path": "papers/repair/6708c2914af1d749a2c4.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1505.00521.pdf",
      "bytes": 967992,
      "reused": false,
      "local_path": "papers/repair/b650ae834f1aaa9996e5.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1511.06392.pdf",
      "bytes": 433861,
      "reused": false,
      "local_path": "papers/repair/fd4a1acc99eee487cc4e.pdf"
    },
    {
      "status": "ok",
      "url": "http://papers.nips.cc/paper/5648-learning-to-transduce-with-unbounded-memory.pdf",
      "bytes": 587825,
      "reused": false,
      "local_path": "papers/repair/047e9d3801a695c3d95f.pdf"
    },
    {
      "status": "ok",
      "url": "http://papers.nips.cc/paper/5857-inferring-algorithmic-patterns-with-stack-augmented-recurrent-nets.pdf",
      "bytes": 652510,
      "reused": false,
      "local_path": "papers/repair/0c62c20d8d8c62e2781a.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1410.3916.pdf",
      "bytes": 215900,
      "reused": false,
      "local_path": "papers/repair/95ed67d5fa02ade17abb.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1506.07285.pdf",
      "bytes": 923918,
      "reused": false,
      "local_path": "papers/repair/fc32ae3a2c6bcae5f2f9.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1508.01211.pdf",
      "bytes": 2290560,
      "reused": false,
      "local_path": "papers/repair/d18e613ea1faf54b3a00.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1506.05869.pdf",
      "bytes": 89422,
      "reused": false,
      "local_path": "papers/repair/70c1979c568a7767471d.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1603.08983.pdf",
      "bytes": 6582868,
      "reused": false,
      "local_path": "papers/repair/7ab2fb7b2bddef61d0d9.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1511.04834.pdf",
      "bytes": 577767,
      "reused": false,
      "local_path": "papers/repair/573d63852b4c2dcad254.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1511.06279.pdf",
      "bytes": 604529,
      "reused": false,
      "local_path": "papers/repair/6e742e47d5d9e71296fa.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1606.04442.pdf",
      "bytes": 1233094,
      "reused": false,
      "local_path": "papers/repair/4f9dd707609f0a8b7c49.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1602.03218.pdf",
      "bytes": 177201,
      "reused": false,
      "local_path": "papers/repair/d7999ec7bebecfce139e.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 35146,
  "expected_body_images": 8,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/c809afc5d1f7/pages/source.html",
  "local_html": "attachments/c809afc5d1f7/pages/article.html",
  "body_base_path": "attachments/c809afc5d1f7/pages/",
  "body_markdown_path": "attachments/c809afc5d1f7/pages/article.md"
}