{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "PyTorch Tutorials",
  "title_listed": "Sequence Models and LSTM Networks",
  "keywords": "LSTM,序列模型,RNN",
  "domain": "文本处理",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html",
  "publish_date": "持续更新",
  "corpus_stage": "文本处理",
  "notes": "序列模型与 LSTM",
  "id": "3d147f89c645",
  "collected_at": "2026-10-02T16:54:31",
  "title": "Sequence Models and Long Short-Term Memory Networks — PyTorch Tutorials 2.14.0+cu130 documentation",
  "body_markdown": "Note\nGo to the end\nto download the full example code.\nSequence Models and Long Short-Term Memory Networks\n#\nCreated On: Apr 08, 2017 | Last Updated: Jan 07, 2022 | Last Verified: Not Verified\nAt this point, we have seen various feed-forward networks. That is,\nthere is no state maintained by the network at all. This might not be\nthe behavior we want. Sequence models are central to NLP: they are\nmodels where there is some sort of dependence through time between your\ninputs. The classical example of a sequence model is the Hidden Markov\nModel for part-of-speech tagging. Another example is the conditional\nrandom field.\nA recurrent neural network is a network that maintains some kind of\nstate. For example, its output could be used as part of the next input,\nso that information can propagate along as the network passes over the\nsequence. In the case of an LSTM, for each element in the sequence,\nthere is a corresponding\nhidden state\n\\(h_t\\)\n, which in principle\ncan contain information from arbitrary points earlier in the sequence.\nWe can use the hidden state to predict words in a language model,\npart-of-speech tags, and a myriad of other things.\nLSTMs in Pytorch\n#\nBefore getting to the example, note a few things. Pytorch’s LSTM expects\nall of its inputs to be 3D tensors. The semantics of the axes of these\ntensors is important. The first axis is the sequence itself, the second\nindexes instances in the mini-batch, and the third indexes elements of\nthe input. We haven’t discussed mini-batching, so let’s just ignore that\nand assume we will always have just 1 dimension on the second axis. If\nwe want to run the sequence model over the sentence “The cow jumped”,\nour input should look like\n\\[\\begin{bmatrix}\n\\overbrace{q_\\text{The}}^\\text{row vector} \\\\\nq_\\text{cow} \\\\\nq_\\text{jumped}\n\\end{bmatrix}\\]\nExcept remember there is an additional 2nd dimension with size 1.\nIn addition, you could go through the sequence one at a time, in which\ncase the 1st axis will have size 1 also.\nLet’s see a quick example.\n# Author: Robert Guthrie\nimport\ntorch\nimport\ntorch.nn\nas\nnn\nimport\ntorch.nn.functional\nas\nF\nimport\ntorch.optim\nas\noptim\ntorch\n.\nmanual_seed\n(\n1\n)\n<torch._C.Generator object at 0x7f51d9f62090>\nlstm\n=\nnn\n.\nLSTM\n(\n3\n,\n3\n)\n# Input dim is 3, output dim is 3\ninputs\n=\n[\ntorch\n.\nrandn\n(\n1\n,\n3\n)\nfor\n_\nin\nrange\n(\n5\n)]\n# make a sequence of length 5\n# initialize the hidden state.\nhidden\n=\n(\ntorch\n.\nrandn\n(\n1\n,\n1\n,\n3\n),\ntorch\n.\nrandn\n(\n1\n,\n1\n,\n3\n))\nfor\ni\nin\ninputs\n:\n# Step through the sequence one element at a time.\n# after each step, hidden contains the hidden state.\nout\n,\nhidden\n=\nlstm\n(\ni\n.\nview\n(\n1\n,\n1\n,\n-\n1\n),\nhidden\n)\n# alternatively, we can do the entire sequence all at once.\n# the first value returned by LSTM is all of the hidden states throughout\n# the sequence. the second is just the most recent hidden state\n# (compare the last slice of \"out\" with \"hidden\" below, they are the same)\n# The reason for this is that:\n# \"out\" will give you access to all hidden states in the sequence\n# \"hidden\" will allow you to continue the sequence and backpropagate,\n# by passing it as an argument  to the lstm at a later time\n# Add the extra 2nd dimension\ninputs\n=\ntorch\n.\ncat\n(\ninputs\n)\n.\nview\n(\nlen\n(\ninputs\n),\n1\n,\n-\n1\n)\nhidden\n=\n(\ntorch\n.\nrandn\n(\n1\n,\n1\n,\n3\n),\ntorch\n.\nrandn\n(\n1\n,\n1\n,\n3\n))\n# clean out hidden state\nout\n,\nhidden\n=\nlstm\n(\ninputs\n,\nhidden\n)\nprint\n(\nout\n)\nprint\n(\nhidden\n)\ntensor([[[-0.0187,  0.1713, -0.2944]],\n\n        [[-0.3521,  0.1026, -0.2971]],\n\n        [[-0.3191,  0.0781, -0.1957]],\n\n        [[-0.1634,  0.0941, -0.1637]],\n\n        [[-0.3368,  0.0959, -0.0538]]], grad_fn=<MkldnnRnnLayerBackward0>)\n(tensor([[[-0.3368,  0.0959, -0.0538]]], grad_fn=<StackBackward0>), tensor([[[-0.9825,  0.4715, -0.0633]]], grad_fn=<StackBackward0>))\nExample: An LSTM for Part-of-Speech Tagging\n#\nIn this section, we will use an LSTM to get part of speech tags. We will\nnot use Viterbi or Forward-Backward or anything like that, but as a\n(challenging) exercise to the reader, think about how Viterbi could be\nused after you have seen what is going on. In this example, we also refer\nto embeddings. If you are unfamiliar with embeddings, you can read up\nabout them\nhere\n.\nThe model is as follows: let our input sentence be\n\\(w_1, \\dots, w_M\\)\n, where\n\\(w_i \\in V\\)\n, our vocab. Also, let\n\\(T\\)\nbe our tag set, and\n\\(y_i\\)\nthe tag of word\n\\(w_i\\)\n.\nDenote our prediction of the tag of word\n\\(w_i\\)\nby\n\\(\\hat{y}_i\\)\n.\nThis is a structure prediction, model, where our output is a sequence\n\\(\\hat{y}_1, \\dots, \\hat{y}_M\\)\n, where\n\\(\\hat{y}_i \\in T\\)\n.\nTo do the prediction, pass an LSTM over the sentence. Denote the hidden\nstate at timestep\n\\(i\\)\nas\n\\(h_i\\)\n. Also, assign each tag a\nunique index (like how we had word_to_ix in the word embeddings\nsection). Then our prediction rule for\n\\(\\hat{y}_i\\)\nis\n\\[\\hat{y}_i = \\text{argmax}_j \\  (\\log \\text{Softmax}(Ah_i + b))_j\n\n\\]\nThat is, take the log softmax of the affine map of the hidden state,\nand the predicted tag is the tag that has the maximum value in this\nvector. Note this implies immediately that the dimensionality of the\ntarget space of\n\\(A\\)\nis\n\\(|T|\\)\n.\nPrepare data:\ndef\nprepare_sequence\n(\nseq\n,\nto_ix\n):\nidxs\n=\n[\nto_ix\n[\nw\n]\nfor\nw\nin\nseq\n]\nreturn\ntorch\n.\ntensor\n(\nidxs\n,\ndtype\n=\ntorch\n.\nlong\n)\ntraining_data\n=\n[\n# Tags are: DET - determiner; NN - noun; V - verb\n# For example, the word \"The\" is a determiner\n(\n\"The dog ate the apple\"\n.\nsplit\n(),\n[\n\"DET\"\n,\n\"NN\"\n,\n\"V\"\n,\n\"DET\"\n,\n\"NN\"\n]),\n(\n\"Everybody read that book\"\n.\nsplit\n(),\n[\n\"NN\"\n,\n\"V\"\n,\n\"DET\"\n,\n\"NN\"\n])\n]\nword_to_ix\n=\n{}\n# For each words-list (sentence) and tags-list in each tuple of training_data\nfor\nsent\n,\ntags\nin\ntraining_data\n:\nfor\nword\nin\nsent\n:\nif\nword\nnot\nin\nword_to_ix\n:\n# word has not been assigned an index yet\nword_to_ix\n[\nword\n]\n=\nlen\n(\nword_to_ix\n)\n# Assign each word with a unique index\nprint\n(\nword_to_ix\n)\ntag_to_ix\n=\n{\n\"DET\"\n:\n0\n,\n\"NN\"\n:\n1\n,\n\"V\"\n:\n2\n}\n# Assign each tag with a unique index\n# These will usually be more like 32 or 64 dimensional.\n# We will keep them small, so we can see how the weights change as we train.\nEMBEDDING_DIM\n=\n6\nHIDDEN_DIM\n=\n6\n{'The': 0, 'dog': 1, 'ate': 2, 'the': 3, 'apple': 4, 'Everybody': 5, 'read': 6, 'that': 7, 'book': 8}\nCreate the model:\nclass\nLSTMTagger\n(\nnn\n.\nModule\n):\ndef\n__init__\n(\nself\n,\nembedding_dim\n,\nhidden_dim\n,\nvocab_size\n,\ntagset_size\n):\nsuper\n(\nLSTMTagger\n,\nself\n)\n.\n__init__\n()\nself\n.\nhidden_dim\n=\nhidden_dim\nself\n.\nword_embeddings\n=\nnn\n.\nEmbedding\n(\nvocab_size\n,\nembedding_dim\n)\n# The LSTM takes word embeddings as inputs, and outputs hidden states\n# with dimensionality hidden_dim.\nself\n.\nlstm\n=\nnn\n.\nLSTM\n(\nembedding_dim\n,\nhidden_dim\n)\n# The linear layer that maps from hidden state space to tag space\nself\n.\nhidden2tag\n=\nnn\n.\nLinear\n(\nhidden_dim\n,\ntagset_size\n)\ndef\nforward\n(\nself\n,\nsentence\n):\nembeds\n=\nself\n.\nword_embeddings\n(\nsentence\n)\nlstm_out\n,\n_\n=\nself\n.\nlstm\n(\nembeds\n.\nview\n(\nlen\n(\nsentence\n),\n1\n,\n-\n1\n))\ntag_space\n=\nself\n.\nhidden2tag\n(\nlstm_out\n.\nview\n(\nlen\n(\nsentence\n),\n-\n1\n))\ntag_scores\n=\nF\n.\nlog_softmax\n(\ntag_space\n,\ndim\n=\n1\n)\nreturn\ntag_scores\nTrain the model:\nmodel\n=\nLSTMTagger\n(\nEMBEDDING_DIM\n,\nHIDDEN_DIM\n,\nlen\n(\nword_to_ix\n),\nlen\n(\ntag_to_ix\n))\nloss_function\n=\nnn\n.\nNLLLoss\n()\noptimizer\n=\noptim\n.\nSGD\n(\nmodel\n.\nparameters\n(),\nlr\n=\n0.1\n)\n# See what the scores are before training\n# Note that element i,j of the output is the score for tag j for word i.\n# Here we don't need to train, so the code is wrapped in torch.no_grad()\nwith\ntorch\n.\nno_grad\n():\ninputs\n=\nprepare_sequence\n(\ntraining_data\n[\n0\n][\n0\n],\nword_to_ix\n)\ntag_scores\n=\nmodel\n(\ninputs\n)\nprint\n(\ntag_scores\n)\nfor\nepoch\nin\nrange\n(\n300\n):\n# again, normally you would NOT do 300 epochs, it is toy data\nfor\nsentence\n,\ntags\nin\ntraining_data\n:\n# Step 1. Remember that Pytorch accumulates gradients.\n# We need to clear them out before each instance\nmodel\n.\nzero_grad\n()\n# Step 2. Get our inputs ready for the network, that is, turn them into\n# Tensors of word indices.\nsentence_in\n=\nprepare_sequence\n(\nsentence\n,\nword_to_ix\n)\ntargets\n=\nprepare_sequence\n(\ntags\n,\ntag_to_ix\n)\n# Step 3. Run our forward pass.\ntag_scores\n=\nmodel\n(\nsentence_in\n)\n# Step 4. Compute the loss, gradients, and update the parameters by\n#  calling optimizer.step()\nloss\n=\nloss_function\n(\ntag_scores\n,\ntargets\n)\nloss\n.\nbackward\n()\noptimizer\n.\nstep\n()\n# See what the scores are after training\nwith\ntorch\n.\nno_grad\n():\ninputs\n=\nprepare_sequence\n(\ntraining_data\n[\n0\n][\n0\n],\nword_to_ix\n)\ntag_scores\n=\nmodel\n(\ninputs\n)\n# The sentence is \"the dog ate the apple\".  i,j corresponds to score for tag j\n# for word i. The predicted tag is the maximum scoring tag.\n# Here, we can see the predicted sequence below is 0 1 2 0 1\n# since 0 is index of the maximum value of row 1,\n# 1 is the index of maximum value of row 2, etc.\n# Which is DET NOUN VERB DET NOUN, the correct sequence!\nprint\n(\ntag_scores\n)\ntensor([[-1.1389, -1.2024, -0.9693],\n        [-1.1065, -1.2200, -0.9834],\n        [-1.1286, -1.2093, -0.9726],\n        [-1.1190, -1.1960, -0.9916],\n        [-1.0137, -1.2642, -1.0366]])\ntensor([[-0.0462, -4.0106, -3.6096],\n        [-4.8205, -0.0286, -3.9045],\n        [-3.7876, -4.1355, -0.0394],\n        [-0.0185, -4.7874, -4.6013],\n        [-5.7881, -0.0186, -4.1778]])\nExercise: Augmenting the LSTM part-of-speech tagger with character-level features\n#\nIn the example above, each word had an embedding, which served as the\ninputs to our sequence model. Let’s augment the word embeddings with a\nrepresentation derived from the characters of the word. We expect that\nthis should help significantly, since character-level information like\naffixes have a large bearing on part-of-speech. For example, words with\nthe affix\n-ly\nare almost always tagged as adverbs in English.\nTo do this, let\n\\(c_w\\)\nbe the character-level representation of\nword\n\\(w\\)\n. Let\n\\(x_w\\)\nbe the word embedding as before. Then\nthe input to our sequence model is the concatenation of\n\\(x_w\\)\nand\n\\(c_w\\)\n. So if\n\\(x_w\\)\nhas dimension 5, and\n\\(c_w\\)\ndimension 3, then our LSTM should accept an input of dimension 8.\nTo get the character level representation, do an LSTM over the\ncharacters of a word, and let\n\\(c_w\\)\nbe the final hidden state of\nthis LSTM. Hints:\nThere are going to be two LSTM’s in your new model.\nThe original one that outputs POS tag scores, and the new one that\noutputs a character-level representation of each word.\nTo do a sequence model over characters, you will have to embed characters.\nThe character embeddings will be the input to the character LSTM.\nTotal running time of the script:\n(0 minutes 0.624 seconds)\nDownload\nJupyter\nnotebook:\nsequence_models_tutorial.ipynb\nDownload\nPython\nsource\ncode:\nsequence_models_tutorial.py\nDownload\nzipped:\nsequence_models_tutorial.zip",
  "metadata": {
    "title": "Sequence Models and Long Short-Term Memory Networks — PyTorch Tutorials 2.14.0+cu130 documentation",
    "description": null,
    "keywords": null,
    "author": null,
    "og:image": "https://docs.pytorch.org/docs/stable/_static/img/pytorch_seo.png",
    "article:modified_time": "2022-07-20T23:02:43+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/3d147f89c645/code/body-000.txt",
      "chars": 143,
      "language": "text"
    },
    {
      "file": "attachments/3d147f89c645/code/body-001.txt",
      "chars": 46,
      "language": "text"
    },
    {
      "file": "attachments/3d147f89c645/code/body-002.txt",
      "chars": 1131,
      "language": "text"
    },
    {
      "file": "attachments/3d147f89c645/code/body-003.txt",
      "chars": 370,
      "language": "text"
    },
    {
      "file": "attachments/3d147f89c645/code/body-004.txt",
      "chars": 960,
      "language": "text"
    },
    {
      "file": "attachments/3d147f89c645/code/body-005.txt",
      "chars": 102,
      "language": "text"
    },
    {
      "file": "attachments/3d147f89c645/code/body-006.txt",
      "chars": 868,
      "language": "text"
    },
    {
      "file": "attachments/3d147f89c645/code/body-007.txt",
      "chars": 1844,
      "language": "text"
    },
    {
      "file": "attachments/3d147f89c645/code/body-008.txt",
      "chars": 372,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.manual_seed.html",
      "label": "torch . manual_seed",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.LSTM.html",
      "label": "lstm",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/tensors.html",
      "label": "inputs",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.randn.html",
      "label": "torch . randn",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.cat.html",
      "label": "torch . cat",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.tensor.html",
      "label": "torch . tensor",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/tensor_attributes.html",
      "label": "torch . long",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html",
      "label": "nn . Module",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.Embedding.html",
      "label": "nn . Embedding",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html",
      "label": "nn . Linear",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.log_softmax.html",
      "label": "F . log_softmax",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html",
      "label": "loss_function",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.optim.SGD.html",
      "label": "optimizer",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.no_grad.html",
      "label": "torch . no_grad",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.Tensor.backward.html",
      "label": "loss . backward",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/_downloads/5edaebfc06ec3968b8c1da100da2253d/sequence_models_tutorial.ipynb",
      "label": "Download Jupyter notebook: sequence_models_tutorial.ipynb",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/_downloads/00262d1f472765b291ae3a1a5bf86bae/sequence_models_tutorial.py",
      "label": "Download Python source code: sequence_models_tutorial.py",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/_downloads/1bc7594c183f27776c2eaa0578b0dad4/sequence_models_tutorial.zip",
      "label": "Download zipped: sequence_models_tutorial.zip",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:54:03Z",
  "source_url": "https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 10718,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/3d147f89c645/pages/source.html",
  "local_html": "attachments/3d147f89c645/pages/article.html",
  "body_base_path": "attachments/3d147f89c645/pages/",
  "body_markdown_path": "attachments/3d147f89c645/pages/article.md"
}