{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Priyesh Dave",
  "title_listed": "Day 22: Sequence-to-Sequence Models",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/priyeshdave6/day-22-sequence-to-sequence-models-28m3",
  "publish_date": "2026-09-21",
  "corpus_stage": "",
  "notes": "",
  "id": "9ece8ea82d36",
  "collected_at": "2026-10-02T13:49:19",
  "title": "Day 22: Sequence-to-Sequence Models - DEV Community",
  "body_markdown": "What Are Sequence-to-Sequence Models?\nA sequence-to-sequence model (often called a seq2seq model) is a type of neural network that transforms one sequence into another. In natural language processing (NLP), a \"sequence\" is usually a sentence, a paragraph, or a list of words. Seq2seq models excel when both the input and the output can be different lengths.\nThese models are central to tasks like machine translation (turning an English sentence into a French sentence), text summarization (condensing a paragraph into a short summary), and building conversational agents (turning a question into a relevant reply). Unlike simple classification, which produces just a single label per input, these tasks require outputs that are also sequences, sometimes longer or shorter than the input.\nSeq2seq models are designed specifically for cases where you need to map entire sequences of any length to corresponding output sequences, which can also vary in length.\nWhy Do We Need Sequence Models?\nCompare two tasks:\nIn sentiment analysis, you read a whole review and output a single answer like “positive” or “negative.” Inputs might be different lengths, but the output is always a single label. Feed-forward or convolutional neural networks can usually handle this by padding or truncating inputs so they’re always a fixed size, then producing a fixed-size output vector.\nNow look at machine translation. Both the input (“I like cats”) and output (“J’aime les chats”) are sequences, and their lengths are unrelated. A fixed-size network can’t naturally output a sentence of arbitrary length. Padding outputs so they’re all the same length isn’t practical—you may end up wasting space, or worse, cutting sentences short.\nTasks like translation, summarization, dialog, or code generation require systems that:\nProcess inputs of any length\nGenerate outputs of any length\nPreserve relationships across both sequences\nSeq2seq models solve these problems by letting networks handle full input and output sequences, without fixed sizes, while learning how each part of the input relates to the output.\nHow Do Seq2Seq Models Work?\nThe core seq2seq model has two main parts: an\nencoder\nand a\ndecoder\n.\nEncoder\n: The encoder reads the input sequence one token at a time (a token could be a character, word, or subword unit). Each token is first mapped to a numeric vector (an embedding). As tokens are read, the encoder maintains a\nhidden state\n—a summary of what’s been seen so far. After the last token, the encoder outputs a set of numbers called the\ncontext vector\n. In LSTMs, this means the last hidden state and the last cell state. This context is a summary of the entire input.\nDecoder\n: The decoder receives the context vector and generates an output sequence, one token at a time. At each step, it uses the context from the encoder\nand\nwhat it has already generated. This allows it to decide the next token based on both what it knows about the input and what it has produced so far.\nHere's the basic process, step by step:\nEach input token is converted to an embedding (a vector).\nThe encoder processes these vectors one by one, updating its hidden state after every input.\nWhen finished, the encoder passes its final hidden (and cell) state to the decoder—that’s the context vector.\nThe decoder starts with this initial state and a special start-of-sequence token (\n<sos>\n). It predicts the next output token. After each prediction, that token becomes input for the next step, until an end-of-sequence token (\n<eos>\n) is produced or a maximum length is reached.\nThis architecture means the model can flexibly handle sequences of any reasonable length. While modern models often use \"attention\" to further improve performance, the fundamental encode-then-decode pattern remains the same.\nCore Components: Encoder and Decoder\nEncoder:\nReads each token and converts it to a numeric vector.\nMaintains a hidden state—a set of numbers summarizing all tokens seen so far.\nAfter the last token, produces a final set of hidden (and cell, in LSTMs) states as the\ncontext vector\n. This context summarizes the input as compactly as possible.\nDecoder:\nReceives the context vector from the encoder.\nGenerates the output sequence one token at a time.\nTakes as input the last predicted token (starting with\n<sos>\n) and its own previous hidden and cell state.\nStops generation when it produces the\n<eos>\ntoken.\nA simple example:\nInput: \"ABC\"\nThe encoder converts 'A', 'B', and 'C' to vectors and updates its state with each.\nAfter 'C', its final state is a summary of \"ABC\".\nThe decoder uses that summary to generate 'C', then 'B', then 'A', taking its own previous output as input for the next prediction. It continues until it generates\n<eos>\n.\nAt every decoder step, both the summary context and the tokens generated so far influence the next output. This stepwise process from a fixed summary is the foundation of seq2seq modeling.\nMinimal Seq2Seq Example: Reversing Sequences\nLet’s build a small seq2seq model using TensorFlow and Keras. This toy model reverses sequences of characters: given\n['A', 'B', 'C']\n, it outputs\n['C', 'B', 'A']\n. This illustrates the seq2seq pattern with minimal code.\nimport\nnumpy\nas\nnp\nimport\ntensorflow\nas\ntf\nfrom\ntensorflow.keras\nimport\nlayers\n,\nModel\n# Vocabulary and helpers\nvocab\n=\n[\n'\n<pad>\n'\n,\n'\n<sos>\n'\n,\n'\n<eos>\n'\n]\n+\nlist\n(\n\"\nABCDE\n\"\n)\nvocab_size\n=\nlen\n(\nvocab\n)\nchar_to_idx\n=\n{\nch\n:\nidx\nfor\nidx\n,\nch\nin\nenumerate\n(\nvocab\n)}\nidx_to_char\n=\n{\nidx\n:\nch\nfor\nch\n,\nidx\nin\nchar_to_idx\n.\nitems\n()}\ndef\nencode_seq\n(\nseq\n):\n# Add start (<sos>) and end (<eos>) tokens\nreturn\n[\nchar_to_idx\n[\n'\n<sos>\n'\n]]\n+\n[\nchar_to_idx\n[\nc\n]\nfor\nc\nin\nseq\n]\n+\n[\nchar_to_idx\n[\n'\n<eos>\n'\n]]\ndef\npad_seq\n(\nseq\n,\nmax_len\n):\n# Pad with '<pad>' (index 0) to uniform length\nreturn\nseq\n+\n[\nchar_to_idx\n[\n'\n<pad>\n'\n]]\n*\n(\nmax_len\n-\nlen\n(\nseq\n))\ninputs\n=\n[\n'\nA\n'\n,\n'\nAB\n'\n,\n'\nABC\n'\n,\n'\nABCD\n'\n,\n'\nABCDE\n'\n]\ninput_seqs\n=\n[\nencode_seq\n(\nseq\n)\nfor\nseq\nin\ninputs\n]\ntarget_seqs\n=\n[\nencode_seq\n(\nseq\n[::\n-\n1\n])\nfor\nseq\nin\ninputs\n]\n# reversed\nmax_len\n=\nmax\n(\nlen\n(\nseq\n)\nfor\nseq\nin\ninput_seqs\n+\ntarget_seqs\n)\nX\n=\nnp\n.\narray\n([\npad_seq\n(\nseq\n,\nmax_len\n)\nfor\nseq\nin\ninput_seqs\n])\nY\n=\nnp\n.\narray\n([\npad_seq\n(\nseq\n,\nmax_len\n)\nfor\nseq\nin\ntarget_seqs\n])\n# Model parameters\nembed_dim\n=\n8\nhidden_dim\n=\n16\n# Encoder\nencoder_inputs\n=\nlayers\n.\nInput\n(\nshape\n=\n(\nmax_len\n,))\nencoder_embedding\n=\nlayers\n.\nEmbedding\n(\ninput_dim\n=\nvocab_size\n,\noutput_dim\n=\nembed_dim\n,\nmask_zero\n=\nTrue\n)\nx\n=\nencoder_embedding\n(\nencoder_inputs\n)\nencoder_lstm\n=\nlayers\n.\nLSTM\n(\nhidden_dim\n,\nreturn_state\n=\nTrue\n)\nencoder_outputs\n,\nstate_h\n,\nstate_c\n=\nencoder_lstm\n(\nx\n)\nencoder_states\n=\n[\nstate_h\n,\nstate_c\n]\n# Decoder\ndecoder_inputs\n=\nlayers\n.\nInput\n(\nshape\n=\n(\nmax_len\n,))\ndecoder_embedding\n=\nlayers\n.\nEmbedding\n(\ninput_dim\n=\nvocab_size\n,\noutput_dim\n=\nembed_dim\n,\nmask_zero\n=\nTrue\n)\nx_dec\n=\ndecoder_embedding\n(\ndecoder_inputs\n)\ndecoder_lstm\n=\nlayers\n.\nLSTM\n(\nhidden_dim\n,\nreturn_sequences\n=\nTrue\n,\nreturn_state\n=\nTrue\n)\ndecoder_outputs\n,\n_\n,\n_\n=\ndecoder_lstm\n(\nx_dec\n,\ninitial_state\n=\nencoder_states\n)\ndecoder_dense\n=\nlayers\n.\nTimeDistributed\n(\nlayers\n.\nDense\n(\nvocab_size\n,\nactivation\n=\n\"\nsoftmax\n\"\n))\ndecoder_outputs\n=\ndecoder_dense\n(\ndecoder_outputs\n)\n# Assemble model\nmodel\n=\nModel\n([\nencoder_inputs\n,\ndecoder_inputs\n],\ndecoder_outputs\n)\nmodel\n.\ncompile\n(\noptimizer\n=\n\"\nadam\n\"\n,\nloss\n=\n\"\nsparse_categorical_crossentropy\n\"\n,\nmetrics\n=\n[\n\"\naccuracy\n\"\n])\n# Teacher forcing setup\nY_decoder_input\n=\nnp\n.\narray\n([\npad_seq\n([\nchar_to_idx\n[\n'\n<sos>\n'\n]]\n+\nseq\n[\n1\n:],\nmax_len\n)\nfor\nseq\nin\ntarget_seqs\n])\nY_decoder_target\n=\nnp\n.\nexpand_dims\n(\nY\n,\n-\n1\n)\n# shape (batch, seq, 1)\n# Train\nmodel\n.\nfit\n([\nX\n,\nY_decoder_input\n],\nY_decoder_target\n,\nepochs\n=\n300\n,\nverbose\n=\n0\n)\n# Inference setup: encoder model\nencoder_model\n=\nModel\n(\nencoder_inputs\n,\nencoder_states\n)\n# Inference decoder: predict one token at a time\ndecoder_state_input_h\n=\nlayers\n.\nInput\n(\nshape\n=\n(\nhidden_dim\n,))\ndecoder_state_input_c\n=\nlayers\n.\nInput\n(\nshape\n=\n(\nhidden_dim\n,))\ndecoder_states_inputs\n=\n[\ndecoder_state_input_h\n,\ndecoder_state_input_c\n]\ndec_emb_inf_input\n=\nlayers\n.\nInput\n(\nshape\n=\n(\n1\n,))\ndec_emb_inf\n=\ndecoder_embedding\n(\ndec_emb_inf_input\n)\ndec_outputs_inf\n,\nstate_h_inf\n,\nstate_c_inf\n=\ndecoder_lstm\n(\ndec_emb_inf\n,\ninitial_state\n=\ndecoder_states_inputs\n)\ndec_outputs_inf\n=\ndecoder_dense\n(\ndec_outputs_inf\n)\ndecoder_model\n=\nModel\n([\ndec_emb_inf_input\n]\n+\ndecoder_states_inputs\n,\n[\ndec_outputs_inf\n,\nstate_h_inf\n,\nstate_c_inf\n])\ndef\ndecode_sequence\n(\ninput_seq\n):\n# Encode the input as state vectors\nstates_value\n=\nencoder_model\n.\npredict\n(\ninput_seq\n,\nverbose\n=\n0\n)\n# Initial target sequence is '<sos>'\ntarget_seq\n=\nnp\n.\narray\n([[\nchar_to_idx\n[\n'\n<sos>\n'\n]]])\ndecoded\n=\n[]\nfor\n_\nin\nrange\n(\nmax_len\n):\noutput_tokens\n,\nh\n,\nc\n=\ndecoder_model\n.\npredict\n([\ntarget_seq\n]\n+\nstates_value\n,\nverbose\n=\n0\n)\nsampled_token_index\n=\nnp\n.\nargmax\n(\noutput_tokens\n[\n0\n,\n-\n1\n,\n:])\nsampled_char\n=\nidx_to_char\n[\nsampled_token_index\n]\nif\nsampled_char\n==\n'\n<eos>\n'\nor\nlen\n(\ndecoded\n)\n>\nmax_len\n:\nbreak\ndecoded\n.\nappend\n(\nsampled_char\n)\ntarget_seq\n=\nnp\n.\narray\n([[\nsampled_token_index\n]])\nstates_value\n=\n[\nh\n,\nc\n]\nreturn\n''\n.\njoin\n(\ndecoded\n)\n# Run a few test sequences\nfor\ntest_seq\nin\n[\n'\nA\n'\n,\n'\nAB\n'\n,\n'\nABC\n'\n]:\ninp\n=\nnp\n.\narray\n([\npad_seq\n(\nencode_seq\n(\ntest_seq\n),\nmax_len\n)])\nprint\n(\nf\n\"\nInput:\n{\ntest_seq\n}\n-> Predicted:\n{\ndecode_sequence\n(\ninp\n)\n}\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nHow this works:\nThe encoder ingests a padded, tokenized input (with padding values masked so they’re ignored during computation).\nThe decoder is trained with\nteacher forcing\n: during training, it is fed the actual previous target token at each step, not its own previous guess. This accelerates learning by giving it the true recent history.\nDuring inference, the decoder has to generate each token itself: its last output becomes the input for the next prediction, repeating until\n<eos>\nor a maximum length is reached.\nThe\nLSTM hidden state\n(\nstate_h\n) is its current \"memory\"—what it remembers about the sequence so far. The\ncell state\n(\nstate_c\n) carries longer-term information to help deal with long sequences.\nEmbedding layers translate tokens (like 'A') into small numeric vectors that the LSTM can process more effectively.\nPadding allows us to batch sequences of different lengths in a way libraries like Keras can handle, while masking prevents the model from learning to “predict the padding.”\nThis code demonstrates the core seq2seq workflow:\nThe encoder summarizes an arbitrary-length input into a fixed set of numbers.\nThe decoder generates an arbitrary-length output, one token at a time, based on that context and what’s been produced so far.\nMost real-world NLP systems build on these same principles, adding layers like attention to handle longer and more complex data. The fundamental workflow—encoding a flexible input, then decoding a flexible output, one step at a time—remains the backbone of sequence-to-sequence modeling.\nKey Takeaways\nSeq2seq models map input sequences to output sequences of arbitrary lengths.\nEncoder networks summarize input sequences into context vectors.\nDecoder networks generate output sequences step by step using context and prior outputs.\nTeacher forcing during training accelerates learning by providing true previous tokens.\nVariable-length batching is handled with padding and masking in practical implementations.\nTry It Yourself\nWrite a function that takes a list of word strings (e.g., ['cat', 'dog', 'rat']) and returns a new list where each word’s characters are reversed (['tac', 'god', 'tar']). Optionally, adapt the seq2seq example code to use your own custom token pairs for both input and target data. This simulates preparing data for a simple character-level sequence modeling task.\nFurther Resources\n🎥\nEncoder-Decoder Architecture Explained: Seq2Seq Models for Beginners\n📘\nNLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials\n📄\nAn introduction to sequence-to-sequence learning\n📄\n18.1 Encoder‑Decoder Models for Sequence Transduction – Dive into Deep Learning\n📄\nNeural Machine Translation and Sequence‑to‑sequence Models: A Tutorial\nComing up on Day 23:\nThe Encoder-Decoder Architecture",
  "metadata": {
    "title": "Day 22: Sequence-to-Sequence Models - DEV Community",
    "description": "What Are Sequence-to-Sequence Models?   A sequence-to-sequence model (often called a seq2seq... Tagged with ai, nlp, genai, 100daysofcode.",
    "keywords": "ai, nlp, genai, 100daysofcode, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/priyeshdave6/day-22-sequence-to-sequence-models-28m3",
    "og:title": "Day 22: Sequence-to-Sequence Models",
    "og:description": "What Are Sequence-to-Sequence Models?   A sequence-to-sequence model (often called a seq2seq...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx5c87yiowxcm29r11s5y.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Day 22: Sequence-to-Sequence Models",
    "twitter:description": "What Are Sequence-to-Sequence Models?   A sequence-to-sequence model (often called a seq2seq...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx5c87yiowxcm29r11s5y.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/9ece8ea82d36/code/body-000.txt",
      "chars": 3884,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=ye02_9vFr_k",
      "status": "ok",
      "local_path": "attachments/9ece8ea82d36/videos/ye02_9vFr_k.mp4",
      "bytes": 9348910,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "223.941000"
    }
  ],
  "links": [
    {
      "href": "https://www.youtube.com/watch?v=ye02_9vFr_k",
      "label": "Encoder-Decoder Architecture Explained: Seq2Seq Models for Beginners",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html",
      "label": "NLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials",
      "type": "a"
    },
    {
      "href": "https://lorenlugosch.github.io/posts/2019/02/seq2seq/",
      "label": "An introduction to sequence-to-sequence learning",
      "type": "a"
    },
    {
      "href": "https://d2l.smola.org/chapter_natural-language-processing-pretraining/seq2seq.html",
      "label": "18.1 Encoder‑Decoder Models for Sequence Transduction – Dive into Deep Learning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1703.01619",
      "label": "Neural Machine Translation and Sequence‑to‑sequence Models: A Tutorial",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:05Z",
  "source_url": "https://dev.to/priyeshdave6/day-22-sequence-to-sequence-models-28m3",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1703.01619",
      "bytes": 1385349,
      "reused": false,
      "local_path": "papers/repair/3a67bc069bf3e5814248.pdf"
    }
  ],
  "actual_video_files": [
    {
      "local_path": "attachments/9ece8ea82d36/videos/ye02_9vFr_k.mp4",
      "bytes": 9348910,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "223.941000",
      "status": "ok"
    }
  ],
  "body_chars": 12208,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/9ece8ea82d36/pages/source.html",
  "local_html": "attachments/9ece8ea82d36/pages/article.html",
  "body_base_path": "attachments/9ece8ea82d36/pages/",
  "body_markdown_path": "attachments/9ece8ea82d36/pages/article.md"
}