{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "PyTorch Tutorials",
  "title_listed": "NLP from Scratch: Translation with Seq2Seq and Attention",
  "keywords": "Seq2Seq,注意力,翻译",
  "domain": "文本处理",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html",
  "publish_date": "持续更新",
  "corpus_stage": "文本处理",
  "notes": "Seq2Seq 翻译",
  "id": "1f5049cae480",
  "collected_at": "2026-10-02T16:46:22",
  "title": "NLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 2.14.0+cu130 documentation",
  "body_markdown": "Note\nGo to the end\nto download the full example code.\nNLP From Scratch: Translation with a Sequence to Sequence Network and Attention\n#\nCreated On: Mar 24, 2017 | Last Updated: Oct 21, 2024 | Last Verified: Nov 05, 2024\nAuthor\n:\nSean Robertson\nThis tutorials is part of a three-part series:\nNLP From Scratch: Classifying Names with a Character-Level RNN\nNLP From Scratch: Generating Names with a Character-Level RNN\nNLP From Scratch: Translation with a Sequence to Sequence Network and Attention\nThis is the third and final tutorial on doing\nNLP From Scratch\n, where we\nwrite our own classes and functions to preprocess the data to do our NLP\nmodeling tasks.\nIn this project we will be teaching a neural network to translate from\nFrench to English.\n[\nKEY:\n>\ninput,\n=\ntarget,\n<\noutput\n]\n>\nil\nest\nen\ntrain\nde\npeindre\nun\ntableau\n.\n=\nhe\nis\npainting\na\npicture\n.\n<\nhe\nis\npainting\na\npicture\n.\n\n>\npourquoi\nne\npas\nessayer\nce\nvin\ndelicieux\n?\n=\nwhy\nnot\ntry\nthat\ndelicious\nwine\n?\n<\nwhy\nnot\ntry\nthat\ndelicious\nwine\n?\n\n>\nelle\nn\nest\npas\npoete\nmais\nromanciere\n.\n=\nshe\nis\nnot\na\npoet\nbut\na\nnovelist\n.\n<\nshe\nnot\nnot\na\npoet\nbut\na\nnovelist\n.\n\n>\nvous\netes\ntrop\nmaigre\n.\n=\nyou\nre\ntoo\nskinny\n.\n<\nyou\nre\nall\nalone\n.\n… to varying degrees of success.\nThis is made possible by the simple but powerful idea of the\nsequence\nto sequence network\n, in which two\nrecurrent neural networks work together to transform one sequence to\nanother. An encoder network condenses an input sequence into a vector,\nand a decoder network unfolds that vector into a new sequence.\nTo improve upon this model we’ll use an\nattention\nmechanism\n, which lets the decoder\nlearn to focus over a specific range of the input sequence.\nRecommended Reading:\nI assume you have at least installed PyTorch, know Python, and\nunderstand Tensors:\nhttps://pytorch.org/\nFor installation instructions\nDeep Learning with PyTorch: A 60 Minute Blitz\nto get started with PyTorch in general\nLearning PyTorch with Examples\nfor a wide and deep overview\nPyTorch for Former Torch Users\nif you are former Lua Torch user\nIt would also be useful to know about Sequence to Sequence networks and\nhow they work:\nLearning Phrase Representations using RNN Encoder-Decoder for\nStatistical Machine Translation\nSequence to Sequence Learning with Neural\nNetworks\nNeural Machine Translation by Jointly Learning to Align and\nTranslate\nA Neural Conversational Model\nYou will also find the previous tutorials on\nNLP From Scratch: Classifying Names with a Character-Level RNN\nand\nNLP From Scratch: Generating Names with a Character-Level RNN\nhelpful as those concepts are very similar to the Encoder and Decoder\nmodels, respectively.\nRequirements\nfrom\n__future__\nimport\nunicode_literals\n,\nprint_function\n,\ndivision\nfrom\nio\nimport\nopen\nimport\nunicodedata\nimport\nre\nimport\nrandom\nimport\ntorch\nimport\ntorch.nn\nas\nnn\nfrom\ntorch\nimport\noptim\nimport\ntorch.nn.functional\nas\nF\nimport\nnumpy\nas\nnp\nfrom\ntorch.utils.data\nimport\nTensorDataset\n,\nDataLoader\n,\nRandomSampler\ndevice\n=\ntorch\n.\ndevice\n(\n\"cuda\"\nif\ntorch\n.\ncuda\n.\nis_available\n()\nelse\n\"cpu\"\n)\nLoading data files\n#\nThe data for this project is a set of many thousands of English to\nFrench translation pairs.\nThis question on Open Data Stack\nExchange\npointed me to the open translation site\nhttps://tatoeba.org/\nwhich has\ndownloads available at\nhttps://tatoeba.org/eng/downloads\n- and better\nyet, someone did the extra work of splitting language pairs into\nindividual text files here:\nhttps://www.manythings.org/anki/\nThe English to French pairs are too big to include in the repository, so\ndownload to\ndata/eng-fra.txt\nbefore continuing. The file is a tab\nseparated list of translation pairs:\nI\nam\ncold.\nJ\n'\nai\nfroid.\nNote\nDownload the data from\nhere\nand extract it to the current directory.\nSimilar to the character encoding used in the character-level RNN\ntutorials, we will be representing each word in a language as a one-hot\nvector, or giant vector of zeros except for a single one (at the index\nof the word). Compared to the dozens of characters that might exist in a\nlanguage, there are many many more words, so the encoding vector is much\nlarger. We will however cheat a bit and trim the data to only use a few\nthousand words per language.\nWe’ll need a unique index per word to use as the inputs and targets of\nthe networks later. To keep track of all this we will use a helper class\ncalled\nLang\nwhich has word → index (\nword2index\n) and index → word\n(\nindex2word\n) dictionaries, as well as a count of each word\nword2count\nwhich will be used to replace rare words later.\nSOS_token\n=\n0\nEOS_token\n=\n1\nclass\nLang\n:\ndef\n__init__\n(\nself\n,\nname\n):\nself\n.\nname\n=\nname\nself\n.\nword2index\n=\n{}\nself\n.\nword2count\n=\n{}\nself\n.\nindex2word\n=\n{\n0\n:\n\"SOS\"\n,\n1\n:\n\"EOS\"\n}\nself\n.\nn_words\n=\n2\n# Count SOS and EOS\ndef\naddSentence\n(\nself\n,\nsentence\n):\nfor\nword\nin\nsentence\n.\nsplit\n(\n' '\n):\nself\n.\naddWord\n(\nword\n)\ndef\naddWord\n(\nself\n,\nword\n):\nif\nword\nnot\nin\nself\n.\nword2index\n:\nself\n.\nword2index\n[\nword\n]\n=\nself\n.\nn_words\nself\n.\nword2count\n[\nword\n]\n=\n1\nself\n.\nindex2word\n[\nself\n.\nn_words\n]\n=\nword\nself\n.\nn_words\n+=\n1\nelse\n:\nself\n.\nword2count\n[\nword\n]\n+=\n1\nThe files are all in Unicode, to simplify we will turn Unicode\ncharacters to ASCII, make everything lowercase, and trim most\npunctuation.\n# Turn a Unicode string to plain ASCII, thanks to\n# https://stackoverflow.com/a/518232/2809427\ndef\nunicodeToAscii\n(\ns\n):\nreturn\n''\n.\njoin\n(\nc\nfor\nc\nin\nunicodedata\n.\nnormalize\n(\n'NFD'\n,\ns\n)\nif\nunicodedata\n.\ncategory\n(\nc\n)\n!=\n'Mn'\n)\n# Lowercase, trim, and remove non-letter characters\ndef\nnormalizeString\n(\ns\n):\ns\n=\nunicodeToAscii\n(\ns\n.\nlower\n()\n.\nstrip\n())\ns\n=\nre\n.\nsub\n(\nr\n\"([.!?])\"\n,\nr\n\" \\1\"\n,\ns\n)\ns\n=\nre\n.\nsub\n(\nr\n\"[^a-zA-Z!?]+\"\n,\nr\n\" \"\n,\ns\n)\nreturn\ns\n.\nstrip\n()\nTo read the data file we will split the file into lines, and then split\nlines into pairs. The files are all English → Other Language, so if we\nwant to translate from Other Language → English I added the\nreverse\nflag to reverse the pairs.\ndef\nreadLangs\n(\nlang1\n,\nlang2\n,\nreverse\n=\nFalse\n):\nprint\n(\n\"Reading lines...\"\n)\n# Read the file and split into lines\nlines\n=\nopen\n(\n'data/\n%s\n-\n%s\n.txt'\n%\n(\nlang1\n,\nlang2\n),\nencoding\n=\n'utf-8'\n)\n.\n\\\nread\n()\n.\nstrip\n()\n.\nsplit\n(\n'\n\\n\n'\n)\n# Split every line into pairs and normalize\npairs\n=\n[[\nnormalizeString\n(\ns\n)\nfor\ns\nin\nl\n.\nsplit\n(\n'\n\\t\n'\n)]\nfor\nl\nin\nlines\n]\n# Reverse pairs, make Lang instances\nif\nreverse\n:\npairs\n=\n[\nlist\n(\nreversed\n(\np\n))\nfor\np\nin\npairs\n]\ninput_lang\n=\nLang\n(\nlang2\n)\noutput_lang\n=\nLang\n(\nlang1\n)\nelse\n:\ninput_lang\n=\nLang\n(\nlang1\n)\noutput_lang\n=\nLang\n(\nlang2\n)\nreturn\ninput_lang\n,\noutput_lang\n,\npairs\nSince there are a\nlot\nof example sentences and we want to train\nsomething quickly, we’ll trim the data set to only relatively short and\nsimple sentences. Here the maximum length is 10 words (that includes\nending punctuation) and we’re filtering to sentences that translate to\nthe form “I am” or “He is” etc. (accounting for apostrophes replaced\nearlier).\nMAX_LENGTH\n=\n10\neng_prefixes\n=\n(\n\"i am \"\n,\n\"i m \"\n,\n\"he is\"\n,\n\"he s \"\n,\n\"she is\"\n,\n\"she s \"\n,\n\"you are\"\n,\n\"you re \"\n,\n\"we are\"\n,\n\"we re \"\n,\n\"they are\"\n,\n\"they re \"\n)\ndef\nfilterPair\n(\np\n):\nreturn\nlen\n(\np\n[\n0\n]\n.\nsplit\n(\n' '\n))\n<\nMAX_LENGTH\nand\n\\\nlen\n(\np\n[\n1\n]\n.\nsplit\n(\n' '\n))\n<\nMAX_LENGTH\nand\n\\\np\n[\n1\n]\n.\nstartswith\n(\neng_prefixes\n)\ndef\nfilterPairs\n(\npairs\n):\nreturn\n[\npair\nfor\npair\nin\npairs\nif\nfilterPair\n(\npair\n)]\nThe full process for preparing the data is:\nRead text file and split into lines, split lines into pairs\nNormalize text, filter by length and content\nMake word lists from sentences in pairs\ndef\nprepareData\n(\nlang1\n,\nlang2\n,\nreverse\n=\nFalse\n):\ninput_lang\n,\noutput_lang\n,\npairs\n=\nreadLangs\n(\nlang1\n,\nlang2\n,\nreverse\n)\nprint\n(\n\"Read\n%s\nsentence pairs\"\n%\nlen\n(\npairs\n))\npairs\n=\nfilterPairs\n(\npairs\n)\nprint\n(\n\"Trimmed to\n%s\nsentence pairs\"\n%\nlen\n(\npairs\n))\nprint\n(\n\"Counting words...\"\n)\nfor\npair\nin\npairs\n:\ninput_lang\n.\naddSentence\n(\npair\n[\n0\n])\noutput_lang\n.\naddSentence\n(\npair\n[\n1\n])\nprint\n(\n\"Counted words:\"\n)\nprint\n(\ninput_lang\n.\nname\n,\ninput_lang\n.\nn_words\n)\nprint\n(\noutput_lang\n.\nname\n,\noutput_lang\n.\nn_words\n)\nreturn\ninput_lang\n,\noutput_lang\n,\npairs\ninput_lang\n,\noutput_lang\n,\npairs\n=\nprepareData\n(\n'eng'\n,\n'fra'\n,\nTrue\n)\nprint\n(\nrandom\n.\nchoice\n(\npairs\n))\nReading lines...\nRead 135842 sentence pairs\nTrimmed to 11445 sentence pairs\nCounting words...\nCounted words:\nfra 4601\neng 2991\n['je vais sortir', 'i m about to go out']\nThe Seq2Seq Model\n#\nA Recurrent Neural Network, or RNN, is a network that operates on a\nsequence and uses its own output as input for subsequent steps.\nA\nSequence to Sequence network\n, or\nseq2seq network, or\nEncoder Decoder\nnetwork\n, is a model\nconsisting of two RNNs called the encoder and decoder. The encoder reads\nan input sequence and outputs a single vector, and the decoder reads\nthat vector to produce an output sequence.\nUnlike sequence prediction with a single RNN, where every input\ncorresponds to an output, the seq2seq model frees us from sequence\nlength and order, which makes it ideal for translation between two\nlanguages.\nConsider the sentence\nJe\nne\nsuis\npas\nle\nchat\nnoir\n→\nI\nam\nnot\nthe\nblack\ncat\n. Most of the words in the input sentence have a direct\ntranslation in the output sentence, but are in slightly different\norders, e.g.\nchat\nnoir\nand\nblack\ncat\n. Because of the\nne/pas\nconstruction there is also one more word in the input sentence. It would\nbe difficult to produce a correct translation directly from the sequence\nof input words.\nWith a seq2seq model the encoder creates a single vector which, in the\nideal case, encodes the “meaning” of the input sequence into a single\nvector — a single point in some N dimensional space of sentences.\nThe Encoder\n#\nThe encoder of a seq2seq network is a RNN that outputs some value for\nevery word from the input sentence. For every input word the encoder\noutputs a vector and a hidden state, and uses the hidden state for the\nnext input word.\nclass\nEncoderRNN\n(\nnn\n.\nModule\n):\ndef\n__init__\n(\nself\n,\ninput_size\n,\nhidden_size\n,\ndropout_p\n=\n0.1\n):\nsuper\n(\nEncoderRNN\n,\nself\n)\n.\n__init__\n()\nself\n.\nhidden_size\n=\nhidden_size\nself\n.\nembedding\n=\nnn\n.\nEmbedding\n(\ninput_size\n,\nhidden_size\n)\nself\n.\ngru\n=\nnn\n.\nGRU\n(\nhidden_size\n,\nhidden_size\n,\nbatch_first\n=\nTrue\n)\nself\n.\ndropout\n=\nnn\n.\nDropout\n(\ndropout_p\n)\ndef\nforward\n(\nself\n,\ninput\n):\nembedded\n=\nself\n.\ndropout\n(\nself\n.\nembedding\n(\ninput\n))\noutput\n,\nhidden\n=\nself\n.\ngru\n(\nembedded\n)\nreturn\noutput\n,\nhidden\nThe Decoder\n#\nThe decoder is another RNN that takes the encoder output vector(s) and\noutputs a sequence of words to create the translation.\nSimple Decoder\n#\nIn the simplest seq2seq decoder we use only last output of the encoder.\nThis last output is sometimes called the\ncontext vector\nas it encodes\ncontext from the entire sequence. This context vector is used as the\ninitial hidden state of the decoder.\nAt every step of decoding, the decoder is given an input token and\nhidden state. The initial input token is the start-of-string\n<SOS>\ntoken, and the first hidden state is the context vector (the encoder’s\nlast hidden state).\nclass\nDecoderRNN\n(\nnn\n.\nModule\n):\ndef\n__init__\n(\nself\n,\nhidden_size\n,\noutput_size\n):\nsuper\n(\nDecoderRNN\n,\nself\n)\n.\n__init__\n()\nself\n.\nembedding\n=\nnn\n.\nEmbedding\n(\noutput_size\n,\nhidden_size\n)\nself\n.\ngru\n=\nnn\n.\nGRU\n(\nhidden_size\n,\nhidden_size\n,\nbatch_first\n=\nTrue\n)\nself\n.\nout\n=\nnn\n.\nLinear\n(\nhidden_size\n,\noutput_size\n)\ndef\nforward\n(\nself\n,\nencoder_outputs\n,\nencoder_hidden\n,\ntarget_tensor\n=\nNone\n):\nbatch_size\n=\nencoder_outputs\n.\nsize\n(\n0\n)\ndecoder_input\n=\ntorch\n.\nempty\n(\nbatch_size\n,\n1\n,\ndtype\n=\ntorch\n.\nlong\n,\ndevice\n=\ndevice\n)\n.\nfill_\n(\nSOS_token\n)\ndecoder_hidden\n=\nencoder_hidden\ndecoder_outputs\n=\n[]\nfor\ni\nin\nrange\n(\nMAX_LENGTH\n):\ndecoder_output\n,\ndecoder_hidden\n=\nself\n.\nforward_step\n(\ndecoder_input\n,\ndecoder_hidden\n)\ndecoder_outputs\n.\nappend\n(\ndecoder_output\n)\nif\ntarget_tensor\nis\nnot\nNone\n:\n# Teacher forcing: Feed the target as the next input\ndecoder_input\n=\ntarget_tensor\n[:,\ni\n]\n.\nunsqueeze\n(\n1\n)\n# Teacher forcing\nelse\n:\n# Without teacher forcing: use its own predictions as the next input\n_\n,\ntopi\n=\ndecoder_output\n.\ntopk\n(\n1\n)\ndecoder_input\n=\ntopi\n.\nsqueeze\n(\n-\n1\n)\n.\ndetach\n()\n# detach from history as input\ndecoder_outputs\n=\ntorch\n.\ncat\n(\ndecoder_outputs\n,\ndim\n=\n1\n)\ndecoder_outputs\n=\nF\n.\nlog_softmax\n(\ndecoder_outputs\n,\ndim\n=-\n1\n)\nreturn\ndecoder_outputs\n,\ndecoder_hidden\n,\nNone\n# We return `None` for consistency in the training loop\ndef\nforward_step\n(\nself\n,\ninput\n,\nhidden\n):\noutput\n=\nself\n.\nembedding\n(\ninput\n)\noutput\n=\nF\n.\nrelu\n(\noutput\n)\noutput\n,\nhidden\n=\nself\n.\ngru\n(\noutput\n,\nhidden\n)\noutput\n=\nself\n.\nout\n(\noutput\n)\nreturn\noutput\n,\nhidden\nI encourage you to train and observe the results of this model, but to\nsave space we’ll be going straight for the gold and introducing the\nAttention Mechanism.\nAttention Decoder\n#\nIf only the context vector is passed between the encoder and decoder,\nthat single vector carries the burden of encoding the entire sentence.\nAttention allows the decoder network to “focus” on a different part of\nthe encoder’s outputs for every step of the decoder’s own outputs. First\nwe calculate a set of\nattention weights\n. These will be multiplied by\nthe encoder output vectors to create a weighted combination. The result\n(called\nattn_applied\nin the code) should contain information about\nthat specific part of the input sequence, and thus help the decoder\nchoose the right output words.\nCalculating the attention weights is done with another feed-forward\nlayer\nattn\n, using the decoder’s input and hidden state as inputs.\nBecause there are sentences of all sizes in the training data, to\nactually create and train this layer we have to choose a maximum\nsentence length (input length, for encoder outputs) that it can apply\nto. Sentences of the maximum length will use all the attention weights,\nwhile shorter sentences will only use the first few.\nBahdanau attention, also known as additive attention, is a commonly used\nattention mechanism in sequence-to-sequence models, particularly in neural\nmachine translation tasks. It was introduced by Bahdanau et al. in their\npaper titled\nNeural Machine Translation by Jointly Learning to Align and Translate\n.\nThis attention mechanism employs a learned alignment model to compute attention\nscores between the encoder and decoder hidden states. It utilizes a feed-forward\nneural network to calculate alignment scores.\nHowever, there are alternative attention mechanisms available, such as Luong attention,\nwhich computes attention scores by taking the dot product between the decoder hidden\nstate and the encoder hidden states. It does not involve the non-linear transformation\nused in Bahdanau attention.\nIn this tutorial, we will be using Bahdanau attention. However, it would be a valuable\nexercise to explore modifying the attention mechanism to use Luong attention.\nclass\nBahdanauAttention\n(\nnn\n.\nModule\n):\ndef\n__init__\n(\nself\n,\nhidden_size\n):\nsuper\n(\nBahdanauAttention\n,\nself\n)\n.\n__init__\n()\nself\n.\nWa\n=\nnn\n.\nLinear\n(\nhidden_size\n,\nhidden_size\n)\nself\n.\nUa\n=\nnn\n.\nLinear\n(\nhidden_size\n,\nhidden_size\n)\nself\n.\nVa\n=\nnn\n.\nLinear\n(\nhidden_size\n,\n1\n)\ndef\nforward\n(\nself\n,\nquery\n,\nkeys\n):\nscores\n=\nself\n.\nVa\n(\ntorch\n.\ntanh\n(\nself\n.\nWa\n(\nquery\n)\n+\nself\n.\nUa\n(\nkeys\n)))\nscores\n=\nscores\n.\nsqueeze\n(\n2\n)\n.\nunsqueeze\n(\n1\n)\nweights\n=\nF\n.\nsoftmax\n(\nscores\n,\ndim\n=-\n1\n)\ncontext\n=\ntorch\n.\nbmm\n(\nweights\n,\nkeys\n)\nreturn\ncontext\n,\nweights\nclass\nAttnDecoderRNN\n(\nnn\n.\nModule\n):\ndef\n__init__\n(\nself\n,\nhidden_size\n,\noutput_size\n,\ndropout_p\n=\n0.1\n):\nsuper\n(\nAttnDecoderRNN\n,\nself\n)\n.\n__init__\n()\nself\n.\nembedding\n=\nnn\n.\nEmbedding\n(\noutput_size\n,\nhidden_size\n)\nself\n.\nattention\n=\nBahdanauAttention\n(\nhidden_size\n)\nself\n.\ngru\n=\nnn\n.\nGRU\n(\n2\n*\nhidden_size\n,\nhidden_size\n,\nbatch_first\n=\nTrue\n)\nself\n.\nout\n=\nnn\n.\nLinear\n(\nhidden_size\n,\noutput_size\n)\nself\n.\ndropout\n=\nnn\n.\nDropout\n(\ndropout_p\n)\ndef\nforward\n(\nself\n,\nencoder_outputs\n,\nencoder_hidden\n,\ntarget_tensor\n=\nNone\n):\nbatch_size\n=\nencoder_outputs\n.\nsize\n(\n0\n)\ndecoder_input\n=\ntorch\n.\nempty\n(\nbatch_size\n,\n1\n,\ndtype\n=\ntorch\n.\nlong\n,\ndevice\n=\ndevice\n)\n.\nfill_\n(\nSOS_token\n)\ndecoder_hidden\n=\nencoder_hidden\ndecoder_outputs\n=\n[]\nattentions\n=\n[]\nfor\ni\nin\nrange\n(\nMAX_LENGTH\n):\ndecoder_output\n,\ndecoder_hidden\n,\nattn_weights\n=\nself\n.\nforward_step\n(\ndecoder_input\n,\ndecoder_hidden\n,\nencoder_outputs\n)\ndecoder_outputs\n.\nappend\n(\ndecoder_output\n)\nattentions\n.\nappend\n(\nattn_weights\n)\nif\ntarget_tensor\nis\nnot\nNone\n:\n# Teacher forcing: Feed the target as the next input\ndecoder_input\n=\ntarget_tensor\n[:,\ni\n]\n.\nunsqueeze\n(\n1\n)\n# Teacher forcing\nelse\n:\n# Without teacher forcing: use its own predictions as the next input\n_\n,\ntopi\n=\ndecoder_output\n.\ntopk\n(\n1\n)\ndecoder_input\n=\ntopi\n.\nsqueeze\n(\n-\n1\n)\n.\ndetach\n()\n# detach from history as input\ndecoder_outputs\n=\ntorch\n.\ncat\n(\ndecoder_outputs\n,\ndim\n=\n1\n)\ndecoder_outputs\n=\nF\n.\nlog_softmax\n(\ndecoder_outputs\n,\ndim\n=-\n1\n)\nattentions\n=\ntorch\n.\ncat\n(\nattentions\n,\ndim\n=\n1\n)\nreturn\ndecoder_outputs\n,\ndecoder_hidden\n,\nattentions\ndef\nforward_step\n(\nself\n,\ninput\n,\nhidden\n,\nencoder_outputs\n):\nembedded\n=\nself\n.\ndropout\n(\nself\n.\nembedding\n(\ninput\n))\nquery\n=\nhidden\n.\npermute\n(\n1\n,\n0\n,\n2\n)\ncontext\n,\nattn_weights\n=\nself\n.\nattention\n(\nquery\n,\nencoder_outputs\n)\ninput_gru\n=\ntorch\n.\ncat\n((\nembedded\n,\ncontext\n),\ndim\n=\n2\n)\noutput\n,\nhidden\n=\nself\n.\ngru\n(\ninput_gru\n,\nhidden\n)\noutput\n=\nself\n.\nout\n(\noutput\n)\nreturn\noutput\n,\nhidden\n,\nattn_weights\nNote\nThere are other forms of attention that work around the length\nlimitation by using a relative position approach. Read about “local\nattention” in\nEffective Approaches to Attention-based Neural Machine\nTranslation\n.\nTraining\n#\nPreparing Training Data\n#\nTo train, for each pair we will need an input tensor (indexes of the\nwords in the input sentence) and target tensor (indexes of the words in\nthe target sentence). While creating these vectors we will append the\nEOS token to both sequences.\ndef\nindexesFromSentence\n(\nlang\n,\nsentence\n):\nreturn\n[\nlang\n.\nword2index\n[\nword\n]\nfor\nword\nin\nsentence\n.\nsplit\n(\n' '\n)]\ndef\ntensorFromSentence\n(\nlang\n,\nsentence\n):\nindexes\n=\nindexesFromSentence\n(\nlang\n,\nsentence\n)\nindexes\n.\nappend\n(\nEOS_token\n)\nreturn\ntorch\n.\ntensor\n(\nindexes\n,\ndtype\n=\ntorch\n.\nlong\n,\ndevice\n=\ndevice\n)\n.\nview\n(\n1\n,\n-\n1\n)\ndef\ntensorsFromPair\n(\npair\n):\ninput_tensor\n=\ntensorFromSentence\n(\ninput_lang\n,\npair\n[\n0\n])\ntarget_tensor\n=\ntensorFromSentence\n(\noutput_lang\n,\npair\n[\n1\n])\nreturn\n(\ninput_tensor\n,\ntarget_tensor\n)\ndef\nget_dataloader\n(\nbatch_size\n):\ninput_lang\n,\noutput_lang\n,\npairs\n=\nprepareData\n(\n'eng'\n,\n'fra'\n,\nTrue\n)\nn\n=\nlen\n(\npairs\n)\ninput_ids\n=\nnp\n.\nzeros\n((\nn\n,\nMAX_LENGTH\n),\ndtype\n=\nnp\n.\nint32\n)\ntarget_ids\n=\nnp\n.\nzeros\n((\nn\n,\nMAX_LENGTH\n),\ndtype\n=\nnp\n.\nint32\n)\nfor\nidx\n,\n(\ninp\n,\ntgt\n)\nin\nenumerate\n(\npairs\n):\ninp_ids\n=\nindexesFromSentence\n(\ninput_lang\n,\ninp\n)\ntgt_ids\n=\nindexesFromSentence\n(\noutput_lang\n,\ntgt\n)\ninp_ids\n.\nappend\n(\nEOS_token\n)\ntgt_ids\n.\nappend\n(\nEOS_token\n)\ninput_ids\n[\nidx\n,\n:\nlen\n(\ninp_ids\n)]\n=\ninp_ids\ntarget_ids\n[\nidx\n,\n:\nlen\n(\ntgt_ids\n)]\n=\ntgt_ids\ntrain_data\n=\nTensorDataset\n(\ntorch\n.\nLongTensor\n(\ninput_ids\n)\n.\nto\n(\ndevice\n),\ntorch\n.\nLongTensor\n(\ntarget_ids\n)\n.\nto\n(\ndevice\n))\ntrain_sampler\n=\nRandomSampler\n(\ntrain_data\n)\ntrain_dataloader\n=\nDataLoader\n(\ntrain_data\n,\nsampler\n=\ntrain_sampler\n,\nbatch_size\n=\nbatch_size\n)\nreturn\ninput_lang\n,\noutput_lang\n,\ntrain_dataloader\nTraining the Model\n#\nTo train we run the input sentence through the encoder, and keep track\nof every output and the latest hidden state. Then the decoder is given\nthe\n<SOS>\ntoken as its first input, and the last hidden state of the\nencoder as its first hidden state.\n“Teacher forcing” is the concept of using the real target outputs as\neach next input, instead of using the decoder’s guess as the next input.\nUsing teacher forcing causes it to converge faster but\nwhen the trained\nnetwork is exploited, it may exhibit\ninstability\n.\nYou can observe outputs of teacher-forced networks that read with\ncoherent grammar but wander far from the correct translation -\nintuitively it has learned to represent the output grammar and can “pick\nup” the meaning once the teacher tells it the first few words, but it\nhas not properly learned how to create the sentence from the translation\nin the first place.\nBecause of the freedom PyTorch’s autograd gives us, we can randomly\nchoose to use teacher forcing or not with a simple if statement. Turn\nteacher_forcing_ratio\nup to use more of it.\ndef\ntrain_epoch\n(\ndataloader\n,\nencoder\n,\ndecoder\n,\nencoder_optimizer\n,\ndecoder_optimizer\n,\ncriterion\n):\ntotal_loss\n=\n0\nfor\ndata\nin\ndataloader\n:\ninput_tensor\n,\ntarget_tensor\n=\ndata\nencoder_optimizer\n.\nzero_grad\n()\ndecoder_optimizer\n.\nzero_grad\n()\nencoder_outputs\n,\nencoder_hidden\n=\nencoder\n(\ninput_tensor\n)\ndecoder_outputs\n,\n_\n,\n_\n=\ndecoder\n(\nencoder_outputs\n,\nencoder_hidden\n,\ntarget_tensor\n)\nloss\n=\ncriterion\n(\ndecoder_outputs\n.\nview\n(\n-\n1\n,\ndecoder_outputs\n.\nsize\n(\n-\n1\n)),\ntarget_tensor\n.\nview\n(\n-\n1\n)\n)\nloss\n.\nbackward\n()\nencoder_optimizer\n.\nstep\n()\ndecoder_optimizer\n.\nstep\n()\ntotal_loss\n+=\nloss\n.\nitem\n()\nreturn\ntotal_loss\n/\nlen\n(\ndataloader\n)\nThis is a helper function to print time elapsed and estimated time\nremaining given the current time and progress %.\nimport\ntime\nimport\nmath\ndef\nasMinutes\n(\ns\n):\nm\n=\nmath\n.\nfloor\n(\ns\n/\n60\n)\ns\n-=\nm\n*\n60\nreturn\n'\n%d\nm\n%d\ns'\n%\n(\nm\n,\ns\n)\ndef\ntimeSince\n(\nsince\n,\npercent\n):\nnow\n=\ntime\n.\ntime\n()\ns\n=\nnow\n-\nsince\nes\n=\ns\n/\n(\npercent\n)\nrs\n=\nes\n-\ns\nreturn\n'\n%s\n(-\n%s\n)'\n%\n(\nasMinutes\n(\ns\n),\nasMinutes\n(\nrs\n))\nThe whole training process looks like this:\nStart a timer\nInitialize optimizers and criterion\nCreate set of training pairs\nStart empty losses array for plotting\nThen we call\ntrain\nmany times and occasionally print the progress (%\nof examples, time so far, estimated time) and average loss.\ndef\ntrain\n(\ntrain_dataloader\n,\nencoder\n,\ndecoder\n,\nn_epochs\n,\nlearning_rate\n=\n0.001\n,\nprint_every\n=\n100\n,\nplot_every\n=\n100\n):\nstart\n=\ntime\n.\ntime\n()\nplot_losses\n=\n[]\nprint_loss_total\n=\n0\n# Reset every print_every\nplot_loss_total\n=\n0\n# Reset every plot_every\nencoder_optimizer\n=\noptim\n.\nAdam\n(\nencoder\n.\nparameters\n(),\nlr\n=\nlearning_rate\n)\ndecoder_optimizer\n=\noptim\n.\nAdam\n(\ndecoder\n.\nparameters\n(),\nlr\n=\nlearning_rate\n)\ncriterion\n=\nnn\n.\nNLLLoss\n()\nfor\nepoch\nin\nrange\n(\n1\n,\nn_epochs\n+\n1\n):\nloss\n=\ntrain_epoch\n(\ntrain_dataloader\n,\nencoder\n,\ndecoder\n,\nencoder_optimizer\n,\ndecoder_optimizer\n,\ncriterion\n)\nprint_loss_total\n+=\nloss\nplot_loss_total\n+=\nloss\nif\nepoch\n%\nprint_every\n==\n0\n:\nprint_loss_avg\n=\nprint_loss_total\n/\nprint_every\nprint_loss_total\n=\n0\nprint\n(\n'\n%s\n(\n%d\n%d%%\n)\n%.4f\n'\n%\n(\ntimeSince\n(\nstart\n,\nepoch\n/\nn_epochs\n),\nepoch\n,\nepoch\n/\nn_epochs\n*\n100\n,\nprint_loss_avg\n))\nif\nepoch\n%\nplot_every\n==\n0\n:\nplot_loss_avg\n=\nplot_loss_total\n/\nplot_every\nplot_losses\n.\nappend\n(\nplot_loss_avg\n)\nplot_loss_total\n=\n0\nshowPlot\n(\nplot_losses\n)\nPlotting results\n#\nPlotting is done with matplotlib, using the array of loss values\nplot_losses\nsaved while training.\nimport\nmatplotlib.pyplot\nas\nplt\nplt\n.\nswitch_backend\n(\n'agg'\n)\nimport\nmatplotlib.ticker\nas\nticker\nimport\nnumpy\nas\nnp\ndef\nshowPlot\n(\npoints\n):\nplt\n.\nfigure\n()\nfig\n,\nax\n=\nplt\n.\nsubplots\n()\n# this locator puts ticks at regular intervals\nloc\n=\nticker\n.\nMultipleLocator\n(\nbase\n=\n0.2\n)\nax\n.\nyaxis\n.\nset_major_locator\n(\nloc\n)\nplt\n.\nplot\n(\npoints\n)\nEvaluation\n#\nEvaluation is mostly the same as training, but there are no targets so\nwe simply feed the decoder’s predictions back to itself for each step.\nEvery time it predicts a word we add it to the output string, and if it\npredicts the EOS token we stop there. We also store the decoder’s\nattention outputs for display later.\ndef\nevaluate\n(\nencoder\n,\ndecoder\n,\nsentence\n,\ninput_lang\n,\noutput_lang\n):\nwith\ntorch\n.\nno_grad\n():\ninput_tensor\n=\ntensorFromSentence\n(\ninput_lang\n,\nsentence\n)\nencoder_outputs\n,\nencoder_hidden\n=\nencoder\n(\ninput_tensor\n)\ndecoder_outputs\n,\ndecoder_hidden\n,\ndecoder_attn\n=\ndecoder\n(\nencoder_outputs\n,\nencoder_hidden\n)\n_\n,\ntopi\n=\ndecoder_outputs\n.\ntopk\n(\n1\n)\ndecoded_ids\n=\ntopi\n.\nsqueeze\n()\ndecoded_words\n=\n[]\nfor\nidx\nin\ndecoded_ids\n:\nif\nidx\n.\nitem\n()\n==\nEOS_token\n:\ndecoded_words\n.\nappend\n(\n'<EOS>'\n)\nbreak\ndecoded_words\n.\nappend\n(\noutput_lang\n.\nindex2word\n[\nidx\n.\nitem\n()])\nreturn\ndecoded_words\n,\ndecoder_attn\nWe can evaluate random sentences from the training set and print out the\ninput, target, and output to make some subjective quality judgements:\ndef\nevaluateRandomly\n(\nencoder\n,\ndecoder\n,\nn\n=\n10\n):\nfor\ni\nin\nrange\n(\nn\n):\npair\n=\nrandom\n.\nchoice\n(\npairs\n)\nprint\n(\n'>'\n,\npair\n[\n0\n])\nprint\n(\n'='\n,\npair\n[\n1\n])\noutput_words\n,\n_\n=\nevaluate\n(\nencoder\n,\ndecoder\n,\npair\n[\n0\n],\ninput_lang\n,\noutput_lang\n)\noutput_sentence\n=\n' '\n.\njoin\n(\noutput_words\n)\nprint\n(\n'<'\n,\noutput_sentence\n)\nprint\n(\n''\n)\nTraining and Evaluating\n#\nWith all these helper functions in place (it looks like extra work, but\nit makes it easier to run multiple experiments) we can actually\ninitialize a network and start training.\nRemember that the input sentences were heavily filtered. For this small\ndataset we can use relatively small networks of 256 hidden nodes and a\nsingle GRU layer. After about 40 minutes on a MacBook CPU we’ll get some\nreasonable results.\nNote\nIf you run this notebook you can train, interrupt the kernel,\nevaluate, and continue training later. Comment out the lines where the\nencoder and decoder are initialized and run\ntrainIters\nagain.\nhidden_size\n=\n128\nbatch_size\n=\n32\ninput_lang\n,\noutput_lang\n,\ntrain_dataloader\n=\nget_dataloader\n(\nbatch_size\n)\nencoder\n=\nEncoderRNN\n(\ninput_lang\n.\nn_words\n,\nhidden_size\n)\n.\nto\n(\ndevice\n)\ndecoder\n=\nAttnDecoderRNN\n(\nhidden_size\n,\noutput_lang\n.\nn_words\n)\n.\nto\n(\ndevice\n)\ntrain\n(\ntrain_dataloader\n,\nencoder\n,\ndecoder\n,\n80\n,\nprint_every\n=\n5\n,\nplot_every\n=\n5\n)\nReading lines...\nRead 135842 sentence pairs\nTrimmed to 11445 sentence pairs\nCounting words...\nCounted words:\nfra 4601\neng 2991\n0m 36s (- 9m 4s) (5 6%) 1.5465\n1m 10s (- 8m 16s) (10 12%) 0.6935\n1m 45s (- 7m 37s) (15 18%) 0.3597\n2m 20s (- 7m 0s) (20 25%) 0.1991\n2m 55s (- 6m 25s) (25 31%) 0.1219\n3m 29s (- 5m 49s) (30 37%) 0.0838\n4m 4s (- 5m 14s) (35 43%) 0.0635\n4m 40s (- 4m 40s) (40 50%) 0.0511\n5m 15s (- 4m 5s) (45 56%) 0.0450\n5m 50s (- 3m 30s) (50 62%) 0.0398\n6m 24s (- 2m 54s) (55 68%) 0.0368\n6m 59s (- 2m 19s) (60 75%) 0.0342\n7m 34s (- 1m 44s) (65 81%) 0.0329\n8m 9s (- 1m 9s) (70 87%) 0.0311\n8m 43s (- 0m 34s) (75 93%) 0.0295\n9m 18s (- 0m 0s) (80 100%) 0.0287\nSet dropout layers to\neval\nmode\nencoder\n.\neval\n()\ndecoder\n.\neval\n()\nevaluateRandomly\n(\nencoder\n,\ndecoder\n)\n> il est sous la chaise\n= he s under the chair\n< he s under the chair <EOS>\n\n> nous sommes pretes a y aller\n= we re ready to go\n< we re ready to go <EOS>\n\n> j ecoute la derniere chanson de bjork\n= i m listening to bjork s latest song\n< i m listening to bjork s latest song <EOS>\n\n> je suis epouvantable en math\n= i m terrible at math\n< i m terrible at math <EOS>\n\n> j en ai plutot marre de conduire chaque matin\n= i m getting pretty bored with driving every morning\n< i m getting pretty bored with driving every morning <EOS>\n\n> je crains de ne pas suivre\n= i m afraid i don t follow\n< i m afraid i don t follow any identification <EOS>\n\n> nous sommes desormais maries\n= we re married now\n< we re married now <EOS>\n\n> c est un bon mari pour moi\n= he is a good husband to me\n< he is a good husband to me <EOS>\n\n> tu es en train de perdre\n= you re losing\n< you re losing <EOS>\n\n> je suis fiere de mon pere\n= i m proud of my father\n< i m proud of my father <EOS>\nVisualizing Attention\n#\nA useful property of the attention mechanism is its highly interpretable\noutputs. Because it is used to weight specific encoder outputs of the\ninput sequence, we can imagine looking where the network is focused most\nat each time step.\nYou could simply run\nplt.matshow(attentions)\nto see attention output\ndisplayed as a matrix. For a better viewing experience we will do the\nextra work of adding axes and labels:\ndef\nshowAttention\n(\ninput_sentence\n,\noutput_words\n,\nattentions\n):\nfig\n=\nplt\n.\nfigure\n()\nax\n=\nfig\n.\nadd_subplot\n(\n111\n)\ncax\n=\nax\n.\nmatshow\n(\nattentions\n.\ncpu\n()\n.\nnumpy\n(),\ncmap\n=\n'bone'\n)\nfig\n.\ncolorbar\n(\ncax\n)\n# Set up axes\nax\n.\nset_xticklabels\n([\n''\n]\n+\ninput_sentence\n.\nsplit\n(\n' '\n)\n+\n[\n'<EOS>'\n],\nrotation\n=\n90\n)\nax\n.\nset_yticklabels\n([\n''\n]\n+\noutput_words\n)\n# Show label at every tick\nax\n.\nxaxis\n.\nset_major_locator\n(\nticker\n.\nMultipleLocator\n(\n1\n))\nax\n.\nyaxis\n.\nset_major_locator\n(\nticker\n.\nMultipleLocator\n(\n1\n))\nplt\n.\nshow\n()\ndef\nevaluateAndShowAttention\n(\ninput_sentence\n):\noutput_words\n,\nattentions\n=\nevaluate\n(\nencoder\n,\ndecoder\n,\ninput_sentence\n,\ninput_lang\n,\noutput_lang\n)\nprint\n(\n'input ='\n,\ninput_sentence\n)\nprint\n(\n'output ='\n,\n' '\n.\njoin\n(\noutput_words\n))\nshowAttention\n(\ninput_sentence\n,\noutput_words\n,\nattentions\n[\n0\n,\n:\nlen\n(\noutput_words\n),\n:])\nevaluateAndShowAttention\n(\n'il n est pas aussi grand que son pere'\n)\nevaluateAndShowAttention\n(\n'je suis trop fatigue pour conduire'\n)\nevaluateAndShowAttention\n(\n'je suis desole si c est une question idiote'\n)\nevaluateAndShowAttention\n(\n'je suis reellement fiere de vous'\n)\ninput = il n est pas aussi grand que son pere\noutput = he is not as tall as his father <EOS>\n/var/lib/workspace/intermediate_source/seq2seq_translation_tutorial.py:827: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n  ax.set_xticklabels([''] + input_sentence.split(' ') +\n/var/lib/workspace/intermediate_source/seq2seq_translation_tutorial.py:829: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n  ax.set_yticklabels([''] + output_words)\ninput = je suis trop fatigue pour conduire\noutput = i m too tired to drive jogging <EOS>\n/var/lib/workspace/intermediate_source/seq2seq_translation_tutorial.py:827: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n  ax.set_xticklabels([''] + input_sentence.split(' ') +\n/var/lib/workspace/intermediate_source/seq2seq_translation_tutorial.py:829: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n  ax.set_yticklabels([''] + output_words)\ninput = je suis desole si c est une question idiote\noutput = i m sorry if this is a stupid question <EOS>\n/var/lib/workspace/intermediate_source/seq2seq_translation_tutorial.py:827: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n  ax.set_xticklabels([''] + input_sentence.split(' ') +\n/var/lib/workspace/intermediate_source/seq2seq_translation_tutorial.py:829: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n  ax.set_yticklabels([''] + output_words)\ninput = je suis reellement fiere de vous\noutput = i m really proud of you <EOS>\n/var/lib/workspace/intermediate_source/seq2seq_translation_tutorial.py:827: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n  ax.set_xticklabels([''] + input_sentence.split(' ') +\n/var/lib/workspace/intermediate_source/seq2seq_translation_tutorial.py:829: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n  ax.set_yticklabels([''] + output_words)\nExercises\n#\nTry with a different dataset\nAnother language pair\nHuman → Machine (e.g. IOT commands)\nChat → Response\nQuestion → Answer\nReplace the embeddings with pretrained word embeddings such as\nword2vec\nor\nGloVe\nTry with more layers, more hidden units, and more sentences. Compare\nthe training time and results.\nIf you use a translation file where pairs have two of the same phrase\n(\nI\nam\ntest\n\\t\nI\nam\ntest\n), you can use this as an autoencoder. Try\nthis:\nTrain as an autoencoder\nSave only the Encoder network\nTrain a new Decoder for translation from there\nTotal running time of the script:\n(9 minutes 26.783 seconds)\nDownload\nJupyter\nnotebook:\nseq2seq_translation_tutorial.ipynb\nDownload\nPython\nsource\ncode:\nseq2seq_translation_tutorial.py\nDownload\nzipped:\nseq2seq_translation_tutorial.zip",
  "metadata": {
    "title": "NLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 2.14.0+cu130 documentation",
    "description": null,
    "keywords": null,
    "author": null,
    "og:image": "https://docs.pytorch.org/docs/stable/_static/img/pytorch_seo.png",
    "article:modified_time": "2023-06-09T17:45:43+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/1f5049cae480/code/body-000.txt",
      "chars": 441,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-001.txt",
      "chars": 382,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-002.txt",
      "chars": 26,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-003.txt",
      "chars": 646,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-004.txt",
      "chars": 457,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-005.txt",
      "chars": 641,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-006.txt",
      "chars": 419,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-007.txt",
      "chars": 646,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-008.txt",
      "chars": 169,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-009.txt",
      "chars": 514,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-010.txt",
      "chars": 1687,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-011.txt",
      "chars": 2687,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-012.txt",
      "chars": 1393,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-013.txt",
      "chars": 730,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-014.txt",
      "chars": 284,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-015.txt",
      "chars": 1167,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-016.txt",
      "chars": 332,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-017.txt",
      "chars": 675,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-018.txt",
      "chars": 351,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-019.txt",
      "chars": 318,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-020.txt",
      "chars": 663,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-021.txt",
      "chars": 65,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-022.txt",
      "chars": 960,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-023.txt",
      "chars": 1111,
      "language": "text"
    },
    {
      "file": "attachments/1f5049cae480/code/body-024.txt",
      "chars": 2392,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/seq2seq.png",
      "bytes": 22357,
      "reused": false,
      "local_path": "attachments/1f5049cae480/images/body-233cd225fe129b85cde2.png",
      "src": "https://docs.pytorch.org/tutorials/_images/seq2seq.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/word-encoding.png",
      "bytes": 7887,
      "reused": false,
      "local_path": "attachments/1f5049cae480/images/body-2de707c0dc67c65a650c.png",
      "src": "https://docs.pytorch.org/tutorials/_images/word-encoding.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/encoder-network.png",
      "bytes": 10514,
      "reused": false,
      "local_path": "attachments/1f5049cae480/images/body-66fa06c45e6c8a0983c7.png",
      "src": "https://docs.pytorch.org/tutorials/_images/encoder-network.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/decoder-network.png",
      "bytes": 13423,
      "reused": false,
      "local_path": "attachments/1f5049cae480/images/body-8ce8deab877f407433d8.png",
      "src": "https://docs.pytorch.org/tutorials/_images/decoder-network.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://i.imgur.com/1152PYf.png",
      "bytes": 15600,
      "reused": true,
      "local_path": "attachments/1f5049cae480/images/005.png",
      "src": "https://i.imgur.com/1152PYf.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/attention-decoder-network.png",
      "bytes": 44701,
      "reused": false,
      "local_path": "attachments/1f5049cae480/images/body-df1592cbb8e7629a48e2.png",
      "src": "https://docs.pytorch.org/tutorials/_images/attention-decoder-network.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/sphx_glr_seq2seq_translation_tutorial_001.png",
      "bytes": 2397,
      "reused": false,
      "local_path": "attachments/1f5049cae480/images/body-d97162205532177178b1.png",
      "src": "https://docs.pytorch.org/tutorials/_images/sphx_glr_seq2seq_translation_tutorial_001.png",
      "alt": "seq2seq translation tutorial",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/sphx_glr_seq2seq_translation_tutorial_002.png",
      "bytes": 15802,
      "reused": false,
      "local_path": "attachments/1f5049cae480/images/body-2f97b3b50f4d7ca6cef3.png",
      "src": "https://docs.pytorch.org/tutorials/_images/sphx_glr_seq2seq_translation_tutorial_002.png",
      "alt": "seq2seq translation tutorial",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/sphx_glr_seq2seq_translation_tutorial_003.png",
      "bytes": 18725,
      "reused": false,
      "local_path": "attachments/1f5049cae480/images/body-bca2a4bfccfec55634fd.png",
      "src": "https://docs.pytorch.org/tutorials/_images/sphx_glr_seq2seq_translation_tutorial_003.png",
      "alt": "seq2seq translation tutorial",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/sphx_glr_seq2seq_translation_tutorial_004.png",
      "bytes": 18695,
      "reused": false,
      "local_path": "attachments/1f5049cae480/images/body-e44466f6074aedac7964.png",
      "src": "https://docs.pytorch.org/tutorials/_images/sphx_glr_seq2seq_translation_tutorial_004.png",
      "alt": "seq2seq translation tutorial",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/sphx_glr_seq2seq_translation_tutorial_005.png",
      "bytes": 22314,
      "reused": false,
      "local_path": "attachments/1f5049cae480/images/body-e9a62bfbc2b9c8ec1708.png",
      "src": "https://docs.pytorch.org/tutorials/_images/sphx_glr_seq2seq_translation_tutorial_005.png",
      "alt": "seq2seq translation tutorial",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/sphx_glr_seq2seq_translation_tutorial_006.png",
      "bytes": 18325,
      "reused": false,
      "local_path": "attachments/1f5049cae480/images/body-6778857905b79ad1794a.png",
      "src": "https://docs.pytorch.org/tutorials/_images/sphx_glr_seq2seq_translation_tutorial_006.png",
      "alt": "seq2seq translation tutorial",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/spro",
      "label": "Sean Robertson",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html",
      "label": "NLP From Scratch: Classifying Names with a Character-Level RNN",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html",
      "label": "NLP From Scratch: Generating Names with a Character-Level RNN",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html",
      "label": "NLP From Scratch: Translation with a Sequence to Sequence Network and Attention",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1409.3215",
      "label": "sequence\nto sequence network",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1409.0473",
      "label": "attention\nmechanism",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/",
      "label": "https://pytorch.org/",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html",
      "label": "Deep Learning with PyTorch: A 60 Minute Blitz",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/beginner/pytorch_with_examples.html",
      "label": "Learning PyTorch with Examples",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/beginner/former_torchies_tutorial.html",
      "label": "PyTorch for Former Torch Users",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1406.1078",
      "label": "Learning Phrase Representations using RNN Encoder-Decoder for\nStatistical Machine Translation",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1506.05869",
      "label": "A Neural Conversational Model",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html",
      "label": "NLP From Scratch: Classifying Names with a Character-Level RNN",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html",
      "label": "NLP From Scratch: Generating Names with a Character-Level RNN",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/data.html",
      "label": "TensorDataset",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/tensor_attributes.html",
      "label": "device",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.cuda.is_available.html",
      "label": "torch . cuda . is_available",
      "type": "a"
    },
    {
      "href": "https://opendata.stackexchange.com/questions/3888/dataset-of-sentences-translated-into-many-languages",
      "label": "This question on Open Data Stack\nExchange",
      "type": "a"
    },
    {
      "href": "https://tatoeba.org/",
      "label": "https://tatoeba.org/",
      "type": "a"
    },
    {
      "href": "https://tatoeba.org/eng/downloads",
      "label": "https://tatoeba.org/eng/downloads",
      "type": "a"
    },
    {
      "href": "https://www.manythings.org/anki/",
      "label": "https://www.manythings.org/anki/",
      "type": "a"
    },
    {
      "href": "https://download.pytorch.org/tutorial/data.zip",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1406.1078v3.pdf",
      "label": "Encoder Decoder\nnetwork",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html",
      "label": "nn . Module",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.Embedding.html",
      "label": "nn . Embedding",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.GRU.html",
      "label": "nn . GRU",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.Dropout.html",
      "label": "nn . Dropout",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html",
      "label": "nn . Linear",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.empty.html",
      "label": "torch . empty",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.cat.html",
      "label": "torch . cat",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.log_softmax.html",
      "label": "F . log_softmax",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.relu.html",
      "label": "F . relu",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1409.0473.pdf",
      "label": "Neural Machine Translation by Jointly Learning to Align and Translate",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.tanh.html",
      "label": "torch . tanh",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.softmax.html",
      "label": "F . softmax",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.bmm.html",
      "label": "torch . bmm",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1508.04025",
      "label": "Effective Approaches to Attention-based Neural Machine\nTranslation",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.tensor.html",
      "label": "torch . tensor",
      "type": "a"
    },
    {
      "href": "http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.378.4095&rep=rep1&type=pdf",
      "label": "when the trained\nnetwork is exploited, it may exhibit\ninstability",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.optim.Adam.html",
      "label": "optim . Adam",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html",
      "label": "nn . NLLLoss",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.no_grad.html",
      "label": "torch . no_grad",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/_downloads/032d653a4f5a9c1ec32b9fc7c989ffe1/seq2seq_translation_tutorial.ipynb",
      "label": "Download Jupyter notebook: seq2seq_translation_tutorial.ipynb",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/_downloads/3baf9960a4be104931872ff3ffda07b7/seq2seq_translation_tutorial.py",
      "label": "Download Python source code: seq2seq_translation_tutorial.py",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/_downloads/aa412a8699f8766ba0890a542ef0d9c8/seq2seq_translation_tutorial.zip",
      "label": "Download zipped: seq2seq_translation_tutorial.zip",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:58Z",
  "source_url": "https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1409.3215",
      "bytes": 112084,
      "reused": true,
      "local_path": "papers/repair/61d552395d0f0789263b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1409.0473",
      "bytes": 444482,
      "reused": true,
      "local_path": "papers/repair/b6909588ff7b9ebf375f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1406.1078",
      "bytes": 1142296,
      "reused": true,
      "local_path": "papers/repair/7d6f882033801fb7380b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1506.05869",
      "bytes": 89422,
      "reused": false,
      "local_path": "papers/repair/0f241b7f3c01cb17618d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1406.1078v3.pdf",
      "bytes": 1142296,
      "reused": false,
      "local_path": "papers/repair/d35db0e59682f1a9600c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1409.0473.pdf",
      "bytes": 444482,
      "reused": true,
      "local_path": "papers/repair/ffb5ff16a013792114ac.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1508.04025",
      "bytes": 249825,
      "reused": true,
      "local_path": "papers/repair/94d3d49016ad40753c67.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 32021,
  "expected_body_images": 12,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/1f5049cae480/pages/source.html",
  "local_html": "attachments/1f5049cae480/pages/article.html",
  "body_base_path": "attachments/1f5049cae480/pages/",
  "body_markdown_path": "attachments/1f5049cae480/pages/article.md"
}