{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "Dive into Deep Learning",
  "title_listed": "Recurrent Neural Networks（章节）",
  "keywords": "RNN,LSTM,序列",
  "domain": "文本处理",
  "content_type": "教程/书籍",
  "is_paid_flag": "否",
  "url": "https://www.d2l.ai/chapter_recurrent-neural-networks/index.html",
  "publish_date": "持续更新",
  "corpus_stage": "文本处理",
  "notes": "RNN 章节",
  "id": "dfb87c24dd39",
  "collected_at": "2026-10-02T16:07:41",
  "title": "9. Recurrent Neural Networks — Dive into Deep Learning 1.0.3 documentation",
  "body_markdown": "Table Of Contents\n9.\nRecurrent Neural Networks\n¶\nUp until now, we have focused primarily on fixed-length data. When\nintroducing linear and logistic regression in\nSection 3\nand\nSection 4\nand\nmultilayer perceptrons in\nSection 5\n, we were happy to\nassume that each feature vector\n\\(\\mathbf{x}_i\\)\nconsisted of a\nfixed number of components\n\\(x_1, \\dots, x_d\\)\n, where each numerical\nfeature\n\\(x_j\\)\ncorresponded to a particular attribute. These\ndatasets are sometimes called\ntabular\n, because they can be arranged in\ntables, where each example\n\\(i\\)\ngets its own row, and each\nattribute gets its own column. Crucially, with tabular data, we seldom\nassume any particular structure over the columns.\nSubsequently, in\nSection 7\n, we moved on to image data, where\ninputs consist of the raw pixel values at each coordinate in an image.\nImage data hardly fitted the bill of a protypical tabular dataset.\nThere, we needed to call upon convolutional neural networks (CNNs) to\nhandle the hierarchical structure and invariances. However, our data\nwere still of fixed length. Every Fashion-MNIST image is represented as\na\n\\(28 \\times 28\\)\ngrid of pixel values. Moreover, our goal was to\ndevelop a model that looked at just one image and then outputted a\nsingle prediction. But what should we do when faced with a sequence of\nimages, as in a video, or when tasked with producing a sequentially\nstructured prediction, as in the case of image captioning?\nA great many learning tasks require dealing with sequential data. Image\ncaptioning, speech synthesis, and music generation all require that\nmodels produce outputs consisting of sequences. In other domains, such\nas time series prediction, video analysis, and musical information\nretrieval, a model must learn from inputs that are sequences. These\ndemands often arise simultaneously: tasks such as translating passages\nof text from one natural language to another, engaging in dialogue, or\ncontrolling a robot, demand that models both ingest and output\nsequentially structured data.\nRecurrent neural networks (RNNs) are deep learning models that capture\nthe dynamics of sequences via\nrecurrent\nconnections, which can be\nthought of as cycles in the network of nodes. This might seem\ncounterintuitive at first. After all, it is the feedforward nature of\nneural networks that makes the order of computation unambiguous.\nHowever, recurrent edges are defined in a precise way that ensures that\nno such ambiguity can arise. Recurrent neural networks are\nunrolled\nacross time steps (or sequence steps), with the\nsame\nunderlying\nparameters applied at each step. While the standard connections are\napplied\nsynchronously\nto propagate each layer’s activations to the\nsubsequent layer\nat the same time step\n, the recurrent connections are\ndynamic\n, passing information across adjacent time steps. As the\nunfolded view in\nFig. 9.1\nreveals, RNNs can be\nthought of as feedforward neural networks where each layer’s parameters\n(both conventional and recurrent) are shared across time steps.\nFig. 9.1\nOn the left recurrent connections are depicted via cyclic edges. On\nthe right, we unfold the RNN over time steps. Here, recurrent edges\nspan adjacent time steps, while conventional connections are computed\nsynchronously.\n¶\nLike neural networks more broadly, RNNs have a long discipline-spanning\nhistory, originating as models of the brain popularized by cognitive\nscientists and subsequently adopted as practical modeling tools employed\nby the machine learning community. As we do for deep learning more\nbroadly, in this book we adopt the machine learning perspective,\nfocusing on RNNs as practical tools that rose to popularity in the 2010s\nowing to breakthrough results on such diverse tasks as handwriting\nrecognition\n(\nGraves\net al.\n, 2008\n)\n, machine translation\n(\nSutskever\net al.\n, 2014\n)\n, and recognizing medical diagnoses\n(\nLipton\net al.\n, 2016\n)\n. We point the reader interested in more\nbackground material to a publicly available comprehensive review\n(\nLipton\net al.\n, 2015\n)\n. We also note that sequentiality\nis not unique to RNNs. For example, the CNNs that we already introduced\ncan be adapted to handle data of varying length, e.g., images of varying\nresolution. Moreover, RNNs have recently ceded considerable market share\nto Transformer models, which will be covered in\nSection 11\n. However, RNNs rose to\nprominence as the default models for handling complex sequential\nstructure in deep learning, and remain staple models for sequential\nmodeling to this day. The stories of RNNs and of sequence modeling are\ninextricably linked, and this is as much a chapter about the ABCs of\nsequence modeling problems as it is a chapter about RNNs.\nOne key insight paved the way for a revolution in sequence modeling.\nWhile the inputs and targets for many fundamental tasks in machine\nlearning cannot easily be represented as fixed-length vectors, they can\noften nevertheless be represented as varying-length sequences of\nfixed-length vectors. For example, documents can be represented as\nsequences of words; medical records can often be represented as\nsequences of events (encounters, medications, procedures, lab tests,\ndiagnoses); videos can be represented as varying-length sequences of\nstill images.\nWhile sequence models have popped up in numerous application areas,\nbasic research in the area has been driven predominantly by advances on\ncore tasks in natural language processing. Thus, throughout this\nchapter, we will focus our exposition and examples on text data. If you\nget the hang of these examples, then applying the models to other data\nmodalities should be relatively straightforward. In the next few\nsections, we introduce basic notation for sequences and some evaluation\nmeasures for assessing the quality of sequentially structured model\noutputs. After that, we discuss basic concepts of a language model and\nuse this discussion to motivate our first RNN models. Finally, we\ndescribe the method for calculating gradients when backpropagating\nthrough RNNs and explore some challenges that are often encountered when\ntraining such networks, motivating the modern RNN architectures that\nwill follow in\nSection 10\n.\n9.1. Working with Sequences\n9.1.1. Autoregressive Models\n9.1.2. Sequence Models\n9.1.3. Training\n9.1.4. Prediction\n9.1.5. Summary\n9.1.6. Exercises\n9.2. Converting Raw Text into Sequence Data\n9.2.1. Reading the Dataset\n9.2.2. Tokenization\n9.2.3. Vocabulary\n9.2.4. Putting It All Together\n9.2.5. Exploratory Language Statistics\n9.2.6. Summary\n9.2.7. Exercises\n9.3. Language Models\n9.3.1. Learning Language Models\n9.3.2. Perplexity\n9.3.3. Partitioning Sequences\n9.3.4. Summary and Discussion\n9.3.5. Exercises\n9.4. Recurrent Neural Networks\n9.4.1. Neural Networks without Hidden States\n9.4.2. Recurrent Neural Networks with Hidden States\n9.4.3. RNN-Based Character-Level Language Models\n9.4.4. Summary\n9.4.5. Exercises\n9.5. Recurrent Neural Network Implementation from Scratch\n9.5.1. RNN Model\n9.5.2. RNN-Based Language Model\n9.5.3. Gradient Clipping\n9.5.4. Training\n9.5.5. Decoding\n9.5.6. Summary\n9.5.7. Exercises\n9.6. Concise Implementation of Recurrent Neural Networks\n9.6.1. Defining the Model\n9.6.2. Training and Predicting\n9.6.3. Summary\n9.6.4. Exercises\n9.7. Backpropagation Through Time\n9.7.1. Analysis of Gradients in RNNs\n9.7.2. Backpropagation Through Time in Detail\n9.7.3. Summary\n9.7.4. Exercises\nPrevious\n8.8. Designing Convolution Network Architectures\nNext\n9.1. Working with Sequences",
  "metadata": {
    "title": "9. Recurrent Neural Networks — Dive into Deep Learning 1.0.3 documentation",
    "description": null,
    "keywords": null,
    "author": null
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://www.d2l.ai/_static/logo-with-text.png",
      "bytes": 87175,
      "reused": false,
      "local_path": "attachments/dfb87c24dd39/images/body-509f124ba61493e64845.png",
      "src": "https://www.d2l.ai/_static/logo-with-text.png",
      "alt": "Dive into Deep Learning",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://www.d2l.ai/_images/unfolded-rnn.svg",
      "bytes": 129271,
      "reused": false,
      "local_path": "attachments/dfb87c24dd39/images/body-75f1573de6582942b736.svg",
      "src": "https://www.d2l.ai/_images/unfolded-rnn.svg",
      "alt": "../_images/unfolded-rnn.svg",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.d2l.ai/index.html",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_linear-regression/index.html",
      "label": "Section 3",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_linear-classification/index.html",
      "label": "Section 4",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_multilayer-perceptrons/index.html",
      "label": "Section 5",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_convolutional-neural-networks/index.html",
      "label": "Section 7",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_references/zreferences.html",
      "label": "Graves et al. , 2008",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_attention-mechanisms-and-transformers/index.html",
      "label": "Section 11",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_recurrent-modern/index.html",
      "label": "Section 10",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_recurrent-neural-networks/sequence.html",
      "label": "9.1. Working with Sequences",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_recurrent-neural-networks/text-sequence.html",
      "label": "9.2. Converting Raw Text into Sequence Data",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_recurrent-neural-networks/language-model.html",
      "label": "9.3. Language Models",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_recurrent-neural-networks/rnn.html",
      "label": "9.4. Recurrent Neural Networks",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_recurrent-neural-networks/rnn-scratch.html",
      "label": "9.5. Recurrent Neural Network Implementation from Scratch",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_recurrent-neural-networks/rnn-concise.html",
      "label": "9.6. Concise Implementation of Recurrent Neural Networks",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_recurrent-neural-networks/bptt.html",
      "label": "9.7. Backpropagation Through Time",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_convolutional-modern/cnn-design.html",
      "label": "Previous 8.8. Designing Convolution Network Architectures",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:47Z",
  "source_url": "https://www.d2l.ai/chapter_recurrent-neural-networks/index.html",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 7451,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/dfb87c24dd39/pages/source.html",
  "local_html": "attachments/dfb87c24dd39/pages/article.html",
  "body_base_path": "attachments/dfb87c24dd39/pages/",
  "body_markdown_path": "attachments/dfb87c24dd39/pages/article.md"
}