{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Christopher Olah / colah's blog",
  "title_listed": "Understanding LSTM Networks",
  "keywords": "LSTM,RNN,循环网络",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/",
  "publish_date": "2015-08",
  "corpus_stage": "预训练/SFT",
  "notes": "LSTM网络经典教程，被引用无数",
  "id": "95793dd756cc",
  "collected_at": "2026-10-02T14:26:54",
  "title": "Understanding LSTM Networks -- colah's blog",
  "body_markdown": "Understanding LSTM Networks\nPosted on August 27, 2015\nRecurrent Neural Networks\nHumans don’t start their thinking from scratch every second. As you read this essay, you understand each word based on your understanding of previous words. You don’t throw everything away and start thinking from scratch again. Your thoughts have persistence.\nTraditional neural networks can’t do this, and it seems like a major shortcoming. For example, imagine you want to classify what kind of event is happening at every point in a movie. It’s unclear how a traditional neural network could use its reasoning about previous events in the film to inform later ones.\nRecurrent neural networks address this issue. They are networks with loops in them, allowing information to persist.\nRecurrent Neural Networks have loops.\nIn the above diagram, a chunk of neural network,\n\\(A\\)\n, looks at some input\n\\(x_t\\)\nand outputs a value\n\\(h_t\\)\n. A loop allows information to be passed from one step of the network to the next.\nThese loops make recurrent neural networks seem kind of mysterious. However, if you think a bit more, it turns out that they aren’t all that different than a normal neural network. A recurrent neural network can be thought of as multiple copies of the same network, each passing a message to a successor. Consider what happens if we unroll the loop:\nAn unrolled recurrent neural network.\nThis chain-like nature reveals that recurrent neural networks are intimately related to sequences and lists. They’re the natural architecture of neural network to use for such data.\nAnd they certainly are used! In the last few years, there have been incredible success applying RNNs to a variety of problems: speech recognition, language modeling, translation, image captioning… The list goes on. I’ll leave discussion of the amazing feats one can achieve with RNNs to Andrej Karpathy’s excellent blog post,\nThe Unreasonable Effectiveness of Recurrent Neural Networks\n. But they really are pretty amazing.\nEssential to these successes is the use of “LSTMs,” a very special kind of recurrent neural network which works, for many tasks, much much better than the standard version. Almost all exciting results based on recurrent neural networks are achieved with them. It’s these LSTMs that this essay will explore.\nThe Problem of Long-Term Dependencies\nOne of the appeals of RNNs is the idea that they might be able to connect previous information to the present task, such as using previous video frames might inform the understanding of the present frame. If RNNs could do this, they’d be extremely useful. But can they? It depends.\nSometimes, we only need to look at recent information to perform the present task. For example, consider a language model trying to predict the next word based on the previous ones. If we are trying to predict the last word in “the clouds are in the\nsky\n,” we don’t need any further context – it’s pretty obvious the next word is going to be sky. In such cases, where the gap between the relevant information and the place that it’s needed is small, RNNs can learn to use the past information.\nBut there are also cases where we need more context. Consider trying to predict the last word in the text “I grew up in France… I speak fluent\nFrench\n.” Recent information suggests that the next word is probably the name of a language, but if we want to narrow down which language, we need the context of France, from further back. It’s entirely possible for the gap between the relevant information and the point where it is needed to become very large.\nUnfortunately, as that gap grows, RNNs become unable to learn to connect the information.\nIn theory, RNNs are absolutely capable of handling such “long-term dependencies.” A human could carefully pick parameters for them to solve toy problems of this form. Sadly, in practice, RNNs don’t seem to be able to learn them. The problem was explored in depth by\nHochreiter (1991) [German]\nand\nBengio, et al. (1994)\n, who found some pretty fundamental reasons why it might be difficult.\nThankfully, LSTMs don’t have this problem!\nLSTM Networks\nLong Short Term Memory networks – usually just called “LSTMs” – are a special kind of RNN, capable of learning long-term dependencies. They were introduced by\nHochreiter & Schmidhuber (1997)\n, and were refined and popularized by many people in following work.\n1\nThey work tremendously well on a large variety of problems, and are now widely used.\nLSTMs are explicitly designed to avoid the long-term dependency problem. Remembering information for long periods of time is practically their default behavior, not something they struggle to learn!\nAll recurrent neural networks have the form of a chain of repeating modules of neural network. In standard RNNs, this repeating module will have a very simple structure, such as a single tanh layer.\nThe repeating module in a standard RNN contains a single layer.\nLSTMs also have this chain like structure, but the repeating module has a different structure. Instead of having a single neural network layer, there are four, interacting in a very special way.\nThe repeating module in an LSTM contains four interacting layers.\nDon’t worry about the details of what’s going on. We’ll walk through the LSTM diagram step by step later. For now, let’s just try to get comfortable with the notation we’ll be using.\nIn the above diagram, each line carries an entire vector, from the output of one node to the inputs of others. The pink circles represent pointwise operations, like vector addition, while the yellow boxes are learned neural network layers. Lines merging denote concatenation, while a line forking denote its content being copied and the copies going to different locations.\nThe Core Idea Behind LSTMs\nThe key to LSTMs is the cell state, the horizontal line running through the top of the diagram.\nThe cell state is kind of like a conveyor belt. It runs straight down the entire chain, with only some minor linear interactions. It’s very easy for information to just flow along it unchanged.\nThe LSTM does have the ability to remove or add information to the cell state, carefully regulated by structures called gates.\nGates are a way to optionally let information through. They are composed out of a sigmoid neural net layer and a pointwise multiplication operation.\nThe sigmoid layer outputs numbers between zero and one, describing how much of each component should be let through. A value of zero means “let nothing through,” while a value of one means “let everything through!”\nAn LSTM has three of these gates, to protect and control the cell state.\nStep-by-Step LSTM Walk Through\nThe first step in our LSTM is to decide what information we’re going to throw away from the cell state. This decision is made by a sigmoid layer called the “forget gate layer.” It looks at\n\\(h_{t-1}\\)\nand\n\\(x_t\\)\n, and outputs a number between\n\\(0\\)\nand\n\\(1\\)\nfor each number in the cell state\n\\(C_{t-1}\\)\n. A\n\\(1\\)\nrepresents “completely keep this” while a\n\\(0\\)\nrepresents “completely get rid of this.”\nLet’s go back to our example of a language model trying to predict the next word based on all the previous ones. In such a problem, the cell state might include the gender of the present subject, so that the correct pronouns can be used. When we see a new subject, we want to forget the gender of the old subject.\nThe next step is to decide what new information we’re going to store in the cell state. This has two parts. First, a sigmoid layer called the “input gate layer” decides which values we’ll update. Next, a tanh layer creates a vector of new candidate values,\n\\(\\tilde{C}_t\\)\n, that could be added to the state. In the next step, we’ll combine these two to create an update to the state.\nIn the example of our language model, we’d want to add the gender of the new subject to the cell state, to replace the old one we’re forgetting.\nIt’s now time to update the old cell state,\n\\(C_{t-1}\\)\n, into the new cell state\n\\(C_t\\)\n. The previous steps already decided what to do, we just need to actually do it.\nWe multiply the old state by\n\\(f_t\\)\n, forgetting the things we decided to forget earlier. Then we add\n\\(i_t*\\tilde{C}_t\\)\n. This is the new candidate values, scaled by how much we decided to update each state value.\nIn the case of the language model, this is where we’d actually drop the information about the old subject’s gender and add the new information, as we decided in the previous steps.\nFinally, we need to decide what we’re going to output. This output will be based on our cell state, but will be a filtered version. First, we run a sigmoid layer which decides what parts of the cell state we’re going to output. Then, we put the cell state through\n\\(\\tanh\\)\n(to push the values to be between\n\\(-1\\)\nand\n\\(1\\)\n) and multiply it by the output of the sigmoid gate, so that we only output the parts we decided to.\nFor the language model example, since it just saw a subject, it might want to output information relevant to a verb, in case that’s what is coming next. For example, it might output whether the subject is singular or plural, so that we know what form a verb should be conjugated into if that’s what follows next.\nVariants on Long Short Term Memory\nWhat I’ve described so far is a pretty normal LSTM. But not all LSTMs are the same as the above. In fact, it seems like almost every paper involving LSTMs uses a slightly different version. The differences are minor, but it’s worth mentioning some of them.\nOne popular LSTM variant, introduced by\nGers & Schmidhuber (2000)\n, is adding “peephole connections.” This means that we let the gate layers look at the cell state.\nThe above diagram adds peepholes to all the gates, but many papers will give some peepholes and not others.\nAnother variation is to use coupled forget and input gates. Instead of separately deciding what to forget and what we should add new information to, we make those decisions together. We only forget when we’re going to input something in its place. We only input new values to the state when we forget something older.\nA slightly more dramatic variation on the LSTM is the Gated Recurrent Unit, or GRU, introduced by\nCho, et al. (2014)\n. It combines the forget and input gates into a single “update gate.” It also merges the cell state and hidden state, and makes some other changes. The resulting model is simpler than standard LSTM models, and has been growing increasingly popular.\nThese are only a few of the most notable LSTM variants. There are lots of others, like Depth Gated RNNs by\nYao, et al. (2015)\n. There’s also some completely different approach to tackling long-term dependencies, like Clockwork RNNs by\nKoutnik, et al. (2014)\n.\nWhich of these variants is best? Do the differences matter?\nGreff, et al. (2015)\ndo a nice comparison of popular variants, finding that they’re all about the same.\nJozefowicz, et al. (2015)\ntested more than ten thousand RNN architectures, finding some that worked better than LSTMs on certain tasks.\nConclusion\nEarlier, I mentioned the remarkable results people are achieving with RNNs. Essentially all of these are achieved using LSTMs. They really work a lot better for most tasks!\nWritten down as a set of equations, LSTMs look pretty intimidating. Hopefully, walking through them step by step in this essay has made them a bit more approachable.\nLSTMs were a big step in what we can accomplish with RNNs. It’s natural to wonder: is there another big step? A common opinion among researchers is: “Yes! There is a next step and it’s attention!” The idea is to let every step of an RNN pick information to look at from some larger collection of information. For example, if you are using an RNN to create a caption describing an image, it might pick a part of the image to look at for every word it outputs. In fact,\nXu,\net al.\n(2015)\ndo exactly this – it might be a fun starting point if you want to explore attention! There’s been a number of really exciting results using attention, and it seems like a lot more are around the corner…\nAttention isn’t the only exciting thread in RNN research. For example, Grid LSTMs by\nKalchbrenner,\net al.\n(2015)\nseem extremely promising. Work using RNNs in generative models – such as\nGregor,\net al.\n(2015)\n,\nChung,\net al.\n(2015)\n, or\nBayer & Osendorfer (2015)\n– also seems very interesting. The last few years have been an exciting time for recurrent neural networks, and the coming ones promise to only be more so!\nAcknowledgments\nI’m grateful to a number of people for helping me better understand LSTMs, commenting on the visualizations, and providing feedback on this post.\nI’m very grateful to my colleagues at Google for their helpful feedback, especially\nOriol Vinyals\n,\nGreg Corrado\n,\nJon Shlens\n,\nLuke Vilnis\n, and\nIlya Sutskever\n. I’m also thankful to many other friends and colleagues for taking the time to help me, including\nDario Amodei\n, and\nJacob Steinhardt\n. I’m especially thankful to\nKyunghyun Cho\nfor extremely thoughtful correspondence about my diagrams.\nBefore this post, I practiced explaining LSTMs during two seminar series I taught on neural networks. Thanks to everyone who participated in those for their patience with me, and for their feedback.\nIn addition to the original authors, a lot of people contributed to the modern LSTM. A non-comprehensive list is: Felix Gers, Fred Cummins, Santiago Fernandez, Justin Bayer, Daan Wierstra, Julian Togelius, Faustino Gomez, Matteo Gagliolo, and\nAlex Graves\n.\n↩\nMore Posts\nAttention and Augmented Recurrent Neural Networks\nOn Distill\nConv Nets\nA Modular Perspective\nNeural Networks, Manifolds, and Topology\nDeep Learning, NLP, and Representations\nBuilt by\nOinkina\nwith\nHakyll\nusing\nBootstrap\n,\nMathJax\n,\nDisqus\n,\nMathBox.js\n,\nHighlight.js\n,\n                    and\nFootnotes.js\n.\nEnable JavaScript for footnotes, Disqus comments, and other cool stuff.",
  "metadata": {
    "title": "Understanding LSTM Networks -- colah's blog",
    "description": null,
    "keywords": null,
    "author": null
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-rolled.png",
      "bytes": 21742,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-f6c9eaf57917b3adb275.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-rolled.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png",
      "bytes": 94195,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-8d44d8f9a6a88e11df9f.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png",
      "alt": "An unrolled recurrent neural network.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-shorttermdepdencies.png",
      "bytes": 208106,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-230ba7facd357510491c.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-shorttermdepdencies.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-longtermdependencies.png",
      "bytes": 217756,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-d3660a6c1d896188a5d2.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-longtermdependencies.png",
      "alt": "Neural networks struggle with long term dependencies.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-SimpleRNN.png",
      "bytes": 181021,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-48f9017781d63fcb21fd.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-SimpleRNN.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-chain.png",
      "bytes": 229688,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-1310d44cc068d986b8e6.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-chain.png",
      "alt": "A LSTM neural network.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM2-notation.png",
      "bytes": 24702,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-1de277e5bf1823b0fd3b.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM2-notation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-C-line.png",
      "bytes": 52288,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-fb9bc61ce83b4da0b3b9.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-C-line.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-gate.png",
      "bytes": 5219,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-b0cdf1e48794bc9b2302.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-gate.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-focus-f.png",
      "bytes": 65034,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-6c5fcb3a486085240408.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-focus-f.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-focus-i.png",
      "bytes": 79507,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-451a49fbc0a8b6b7f600.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-focus-i.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-focus-C.png",
      "bytes": 68663,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-a38bda87dea969678442.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-focus-C.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-focus-o.png",
      "bytes": 79589,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-9775cf30b7f480d9b82d.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-focus-o.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-var-peepholes.png",
      "bytes": 101122,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-07951c7c6a6144fc12ab.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-var-peepholes.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-var-tied.png",
      "bytes": 70022,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-919295633209a6f69c17.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-var-tied.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-var-GRU.png",
      "bytes": 94754,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-ccf8d2e59f8c0bb000b2.png",
      "src": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-var-GRU.png",
      "alt": "A gated recurrent unit neural network.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/images/post-covers/attention.png",
      "bytes": 17183,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-d85b97b90d993f890c41.png",
      "src": "https://colah.github.io/images/post-covers/attention.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/fig.png",
      "bytes": 38963,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-4e9777debd9700c02b1f.png",
      "src": "https://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/fig.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/images/post-covers/topology.png",
      "bytes": 175931,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-658044ec699095a4b976.png",
      "src": "https://colah.github.io/images/post-covers/topology.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2014-07-NLP-RNNs-Representations/img/Mikolov-GenderVecs.png",
      "bytes": 11292,
      "reused": false,
      "local_path": "attachments/95793dd756cc/images/body-8edb359b14100ab69ad1.png",
      "src": "https://colah.github.io/posts/2014-07-NLP-RNNs-Representations/img/Mikolov-GenderVecs.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "http://karpathy.github.io/2015/05/21/rnn-effectiveness/",
      "label": "The Unreasonable Effectiveness of Recurrent Neural Networks",
      "type": "a"
    },
    {
      "href": "http://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidhuber.pdf",
      "label": "Hochreiter (1991) [German]",
      "type": "a"
    },
    {
      "href": "http://www-dsi.ing.unifi.it/~paolo/ps/tnn-94-gradient.pdf",
      "label": "Bengio, et al. (1994)",
      "type": "a"
    },
    {
      "href": "http://www.bioinf.jku.at/publications/older/2604.pdf",
      "label": "Hochreiter & Schmidhuber (1997)",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1406.1078v3.pdf",
      "label": "Cho, et al. (2014)",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1508.03790v2.pdf",
      "label": "Yao, et al. (2015)",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1402.3511v1.pdf",
      "label": "Koutnik, et al. (2014)",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1503.04069.pdf",
      "label": "Greff, et al. (2015)",
      "type": "a"
    },
    {
      "href": "http://jmlr.org/proceedings/papers/v37/jozefowicz15.pdf",
      "label": "Jozefowicz, et al. (2015)",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1502.03044v2.pdf",
      "label": "Xu, et al. (2015)",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1507.01526v1.pdf",
      "label": "Kalchbrenner, et al. (2015)",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1502.04623.pdf",
      "label": "Gregor, et al. (2015)",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1506.02216v3.pdf",
      "label": "Chung, et al. (2015)",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1411.7610v3.pdf",
      "label": "Bayer & Osendorfer (2015)",
      "type": "a"
    },
    {
      "href": "http://research.google.com/pubs/OriolVinyals.html",
      "label": "Oriol Vinyals",
      "type": "a"
    },
    {
      "href": "http://research.google.com/pubs/GregCorrado.html",
      "label": "Greg Corrado",
      "type": "a"
    },
    {
      "href": "http://research.google.com/pubs/JonathonShlens.html",
      "label": "Jon Shlens",
      "type": "a"
    },
    {
      "href": "http://people.cs.umass.edu/~luke/",
      "label": "Luke Vilnis",
      "type": "a"
    },
    {
      "href": "http://www.cs.toronto.edu/~ilya/",
      "label": "Ilya Sutskever",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/pub/dario-amodei/4/493/393",
      "label": "Dario Amodei",
      "type": "a"
    },
    {
      "href": "http://cs.stanford.edu/~jsteinhardt/",
      "label": "Jacob Steinhardt",
      "type": "a"
    },
    {
      "href": "http://www.kyunghyuncho.me/",
      "label": "Kyunghyun Cho",
      "type": "a"
    },
    {
      "href": "https://scholar.google.com/citations?user=DaFHynwAAAAJ&hl=en",
      "label": "Alex Graves",
      "type": "a"
    },
    {
      "href": "http://distill.pub/2016/augmented-rnns/",
      "label": "Attention and Augmented Recurrent Neural Networks On Distill",
      "type": "a"
    },
    {
      "href": "https://colah.github.io/posts/2014-07-Conv-Nets-Modular/",
      "label": "Conv Nets A Modular Perspective",
      "type": "a"
    },
    {
      "href": "https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/",
      "label": "Neural Networks, Manifolds, and Topology",
      "type": "a"
    },
    {
      "href": "https://colah.github.io/posts/2014-07-NLP-RNNs-Representations/",
      "label": "Deep Learning, NLP, and Representations",
      "type": "a"
    },
    {
      "href": "https://github.com/oinkina",
      "label": "Oinkina",
      "type": "a"
    },
    {
      "href": "http://jaspervdj.be/hakyll",
      "label": "Hakyll",
      "type": "a"
    },
    {
      "href": "http://getbootstrap.com/",
      "label": "Bootstrap",
      "type": "a"
    },
    {
      "href": "http://www.mathjax.org/",
      "label": "MathJax",
      "type": "a"
    },
    {
      "href": "http://disqus.com/",
      "label": "Disqus",
      "type": "a"
    },
    {
      "href": "https://github.com/unconed/MathBox.js",
      "label": "MathBox.js",
      "type": "a"
    },
    {
      "href": "http://highlightjs.org/",
      "label": "Highlight.js",
      "type": "a"
    },
    {
      "href": "http://ignorethecode.net/blog/2010/04/20/footnotes/",
      "label": "Footnotes.js",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:32Z",
  "source_url": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/",
  "papers": [
    {
      "status": "ok",
      "url": "http://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidhuber.pdf",
      "bytes": 925044,
      "reused": false,
      "local_path": "papers/repair/1c4d0d0ad484b3cd956a.pdf"
    },
    {
      "status": "http_404",
      "url": "http://www-dsi.ing.unifi.it/~paolo/ps/tnn-94-gradient.pdf",
      "retried_at": "2026-10-20T17:36:19Z"
    },
    {
      "status": "ok",
      "url": "http://www.bioinf.jku.at/publications/older/2604.pdf",
      "bytes": 396938,
      "reused": false,
      "local_path": "papers/repair/bf6dc7f68d9932356b86.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1406.1078v3.pdf",
      "bytes": 1142296,
      "reused": false,
      "local_path": "papers/repair/51136433a96ebdb5b3a0.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1508.03790v2.pdf",
      "bytes": 193844,
      "reused": false,
      "local_path": "papers/repair/3beff035dfad1b107781.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1402.3511v1.pdf",
      "bytes": 743435,
      "reused": false,
      "local_path": "papers/repair/fecf819123186dd94091.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1503.04069.pdf",
      "bytes": 2663695,
      "reused": false,
      "local_path": "papers/repair/4a5f3da960e5ff980e26.pdf"
    },
    {
      "status": "http_404",
      "url": "http://jmlr.org/proceedings/papers/v37/jozefowicz15.pdf",
      "retried_at": "2026-10-20T17:36:16Z"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1502.03044v2.pdf",
      "bytes": 9582524,
      "reused": false,
      "local_path": "papers/repair/10c4fbfb0d9ba390969c.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1507.01526v1.pdf",
      "bytes": 802260,
      "reused": false,
      "local_path": "papers/repair/b892ffd7aec8c115a7e0.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1502.04623.pdf",
      "bytes": 1929840,
      "reused": false,
      "local_path": "papers/repair/9d818affc602c28d0878.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1506.02216v3.pdf",
      "bytes": 948399,
      "reused": false,
      "local_path": "papers/repair/afa60c0ef00a708bca9d.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1411.7610v3.pdf",
      "bytes": 2161778,
      "reused": false,
      "local_path": "papers/repair/a903921f60ee76820038.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 13876,
  "expected_body_images": 20,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "http_404",
  "raw_html": "attachments/95793dd756cc/pages/source.html",
  "local_html": "attachments/95793dd756cc/pages/article.html",
  "body_base_path": "attachments/95793dd756cc/pages/",
  "body_markdown_path": "attachments/95793dd756cc/pages/article.md"
}