{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "zeromathai",
  "title_listed": "How Transformer Decoders Generate Text — From Causal Masking to Decoding",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/zeromathai/how-transformer-decoders-generate-text-from-causal-masking-to-decoding-1fh8",
  "publish_date": "2026-06-23",
  "corpus_stage": "",
  "notes": "",
  "id": "9acbf4f536e7",
  "collected_at": "2026-10-02T13:50:42",
  "title": "How Transformer Decoders Generate Text — From Causal Masking to Decoding - DEV Community",
  "body_markdown": "A Transformer Decoder does not generate a sentence all at once.\nIt predicts one token.\nThen it feeds that token back and predicts the next one.\nThat simple loop is the core of modern LLM generation.\nCore Idea\nA Transformer Decoder is built for autoregressive generation.\nThat means:\nprevious tokens → next token prediction → repeat\nThe Decoder creates hidden representations.\nThe LM Head converts those representations into vocabulary scores.\nA decoding strategy chooses the actual next token.\nThis matters because generation quality is not only about the model.\nIt also depends on how tokens are selected.\nThe Key Structure\nA simplified generation pipeline looks like this:\nInput Context\n→ Decoder Layers\n→ Hidden State\n→ LM Head\n→ Logits\n→ Softmax\n→ Decoding Strategy\n→ Next Token\nMore compactly:\nText Generation = decoder representation + vocabulary scoring + token selection\nThe Decoder answers:\nWhat should the next representation be?\nThe LM Head answers:\nWhich vocabulary tokens are likely?\nThe decoding strategy answers:\nWhich token should we actually output?\nPseudo-code View\nAutoregressive decoding looks like this:\ncontext = prompt_tokens\n\nwhile not stop:\n    hidden = decoder(context)\n\n    logits = lm_head(hidden[-1])\n\n    probs = softmax(logits / temperature)\n\n    next_token = decode(probs)\n\n    context.append(next_token)\nEnter fullscreen mode\nExit fullscreen mode\nThe key loop is:\npredict → append → repeat\nThis is why LLM inference is sequential.\nEven if training can be parallelized, generation still produces tokens one step at a time.\nTransformer Decoder Structure\nA Transformer Decoder layer usually contains:\nMasked Self-Attention\nCross-Attention\nFeed-Forward Network\nMasked Self-Attention lets the Decoder look only at previous tokens.\nCross-Attention lets it look at Encoder outputs when an input sequence exists.\nThe Feed-Forward Network transforms each token representation.\nFor decoder-only LLMs, Cross-Attention is usually removed.\nThe model only continues from the current context.\nCausal Masking\nThe Decoder must not cheat.\nWhen predicting token 5, it cannot look at token 6.\nThat is the role of the causal mask.\nThe generation probability can be written as:\nP(y₁, y₂, ..., yₜ | x) = Π P(yₜ | y₁, ..., yₜ₋₁, x)\nEach token depends only on previous output tokens and the input.\nThis is important.\nWithout causal masking, the model could see future answers during training.\nThen it would fail during real generation.\nConcrete Example\nTarget sentence:\nI love you\nDuring training, the Decoder input is shifted right:\nInput:\nI love\nTarget:\nI love you\nSo the model learns:\n→ I\nI → love\nI love → you\nAt inference time, there is no target sentence.\nThe model must use its own previous output.\nThat is why errors can accumulate during generation.\nTeacher Forcing\nTeacher forcing is used during training.\nInstead of feeding the model’s wrong prediction back into the next step, we feed the correct previous token.\nThis makes training more stable.\nTraining:\ninput = correct previous tokens\nEnter fullscreen mode\nExit fullscreen mode\nInference:\ninput = model-generated previous tokens\nEnter fullscreen mode\nExit fullscreen mode\nThis difference matters.\nA model can behave well during training but drift during generation.\nThat is why decoding strategy and evaluation matter in real systems.\nLM Head and Logits\nThe Decoder outputs hidden vectors.\nBut hidden vectors are not tokens.\nThe LM Head maps a hidden vector to vocabulary-sized scores.\nThese scores are called logits.\nIf the vocabulary size is 50,000, the LM Head outputs 50,000 scores.\nEach score corresponds to one possible next token.\nLogits are not probabilities yet.\nSoftmax converts them into probabilities.\nThe pipeline is:\nhidden state → logits → probabilities → selected token\nTemperature Scaling\nTemperature controls how sharp or flat the probability distribution becomes.\nThe formula is:\npᵢ(τ) = exp(zᵢ / τ) / Σ exp(zⱼ / τ)\nLower temperature:\nsharper distribution\nmore deterministic output\nless randomness\nHigher temperature:\nflatter distribution\nmore diverse output\nmore randomness\nExample:\nWith logits [2, 1, 0]:\ntemperature = 0.5 makes the top token much stronger.\ntemperature = 2 makes lower-ranked tokens more likely.\nThis matters in practice.\nTemperature is one of the simplest ways to control creativity.\nWhat Decoding Means\nDecoding means selecting the next token from probabilities.\nThe model gives a distribution.\nThe decoding algorithm makes a choice.\nThat choice affects:\ncorrectness\ncreativity\nrepetition\ndiversity\ndeterminism\nlatency\nSo decoding is not a small detail.\nIt is part of the generation behavior.\nGreedy Decoding\nGreedy decoding always chooses the most likely token.\nIf probabilities are:\nA = 0.70\nB = 0.20\nC = 0.10\nGreedy always picks A.\nIt is simple and fast.\nBut it can be repetitive.\nIt can also choose a locally good token that leads to a worse full sentence.\nBeam Search\nBeam search keeps multiple candidate sequences.\nInstead of only keeping the best next token, it keeps the best k paths.\nIf beam size = 3, the model tracks three candidate continuations.\nThis can improve structured generation.\nBut it can also reduce diversity.\nWhen k = 1, beam search becomes greedy decoding.\nTop-k Sampling\nTop-k sampling keeps only the k most likely tokens.\nThen it samples from that smaller set.\nExample:\nk = 3\nOnly the top 3 tokens can be selected.\nThis prevents the model from choosing extremely unlikely tokens.\nBut it still allows some randomness.\nTop-k is useful when you want controlled diversity.\nTop-p Sampling\nTop-p sampling is also called nucleus sampling.\nInstead of keeping a fixed number of tokens, it keeps the smallest set whose cumulative probability exceeds p.\nExample:\nToken probabilities:\nhoneycomb = 0.45\ngingerbread = 0.20\ndonut = 0.12\ncupcake = 0.04\nIf p = 0.6:\nhoneycomb + gingerbread = 0.65\nSo only those two tokens enter the sampling set.\nTop-p adapts to the confidence of the model.\nThat makes it more flexible than fixed Top-k.\nDeterministic vs Stochastic Decoding\nDeterministic decoding:\ngreedy decoding\nbeam search\nsame input usually gives same output\nuseful for predictable tasks\nStochastic decoding:\nTop-k sampling\nTop-p sampling\ncan generate different outputs\nuseful for creative tasks\nThe difference is simple:\nDeterministic = choose the best-looking path\nStochastic = sample from likely paths\nFor coding tasks, deterministic settings are often useful.\nFor brainstorming, stochastic settings are often better.\nEncoder-Decoder vs Decoder-Only Models\nEncoder-Decoder models use both input understanding and output generation.\nThey are useful for tasks like translation.\nThe Encoder reads the source sequence.\nThe Decoder generates the target sequence.\nDecoder-only models use only the generation stack.\nThey predict the next token from the previous context.\nMost GPT-style LLMs are decoder-only.\nThe architecture is simpler for open-ended text generation.\nImplementation Perspective\nIn real inference code, generation is not just:\nmodel(prompt)\nEnter fullscreen mode\nExit fullscreen mode\nIt is closer to:\ntokenize prompt\n\nrun decoder\n\nget logits from LM Head\n\napply temperature\n\nfilter with top-k or top-p\n\nsample or choose token\n\nappend token\n\nrepeat\nEnter fullscreen mode\nExit fullscreen mode\nThis matters because small decoding changes can produce very different outputs.\nA model can feel precise, boring, creative, unstable, or repetitive depending on decoding settings.\nThe model gives probabilities.\nYour decoding pipeline turns those probabilities into behavior.\nNaive vs Practical View\nNaive view:\nLLM = text in, text out\nPractical view:\nLLM = token loop + logits + decoding policy\nNaive mindset:\nask model\nreceive answer\nEnter fullscreen mode\nExit fullscreen mode\nPractical mindset:\nmanage context\ncontrol temperature\nchoose decoding strategy\nstop generation correctly\nhandle repetition\noptimize inference cost\nEnter fullscreen mode\nExit fullscreen mode\nThis is why developers need to understand the Decoder.\nGeneration is a system, not a single function call.\nImportant Conditions and Limits\nDecoder generation is sequential.\nEach new token depends on previous tokens.\nThat can make inference slow.\nCausal masking is required to prevent future-token leakage.\nTeacher forcing helps training, but inference uses the model’s own predictions.\nDecoding strategy changes output behavior.\nTemperature, Top-k, and Top-p are not cosmetic options.\nThey directly shape the generated text.\nTakeaway\nThe Transformer Decoder generates text by predicting one token at a time.\nMasked Self-Attention prevents future-token access.\nThe LM Head converts hidden states into vocabulary logits.\nSoftmax turns logits into probabilities.\nDecoding chooses the actual next token.\nThe shortest version is:\nDecoder generation = causal attention + LM Head + decoding loop\nIf you understand that loop, you understand how LLMs actually produce text.\nDiscussion\nWhen tuning LLM output, which setting do you usually adjust first?\nTemperature, Top-k, Top-p, or the prompt itself?\nOriginally published at zeromathai.com.\nOriginal article:\nhttps://zeromathai.com/en/transformer-decoder-lm-head-decoding-en/\nGitHub Resources\nAI diagrams, study notes, and visual guides:\nhttps://github.com/zeromathai/zeromathai-ai",
  "metadata": {
    "title": "How Transformer Decoders Generate Text — From Causal Masking to Decoding - DEV Community",
    "description": "A Transformer Decoder does not generate a sentence all at once.  It predicts one token.  Then it... Tagged with ai, machinelearning, llm, deeplearning.",
    "keywords": "ai, machinelearning, llm, deeplearning, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/zeromathai/how-transformer-decoders-generate-text-from-causal-masking-to-decoding-1fh8",
    "og:title": "How Transformer Decoders Generate Text — From Causal Masking to Decoding",
    "og:description": "A Transformer Decoder does not generate a sentence all at once.  It predicts one token.  Then it...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpaj164bbjtxg5ztsxrnu.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "How Transformer Decoders Generate Text — From Causal Masking to Decoding",
    "twitter:description": "A Transformer Decoder does not generate a sentence all at once.  It predicts one token.  Then it...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpaj164bbjtxg5ztsxrnu.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/9acbf4f536e7/code/body-000.txt",
      "chars": 212,
      "language": "text"
    },
    {
      "file": "attachments/9acbf4f536e7/code/body-001.txt",
      "chars": 32,
      "language": "text"
    },
    {
      "file": "attachments/9acbf4f536e7/code/body-002.txt",
      "chars": 40,
      "language": "text"
    },
    {
      "file": "attachments/9acbf4f536e7/code/body-003.txt",
      "chars": 14,
      "language": "text"
    },
    {
      "file": "attachments/9acbf4f536e7/code/body-004.txt",
      "chars": 147,
      "language": "text"
    },
    {
      "file": "attachments/9acbf4f536e7/code/body-005.txt",
      "chars": 25,
      "language": "text"
    },
    {
      "file": "attachments/9acbf4f536e7/code/body-006.txt",
      "chars": 128,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://zeromathai.com/en/transformer-decoder-lm-head-decoding-en/",
      "label": "https://zeromathai.com/en/transformer-decoder-lm-head-decoding-en/",
      "type": "a"
    },
    {
      "href": "https://github.com/zeromathai/zeromathai-ai",
      "label": "https://github.com/zeromathai/zeromathai-ai",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:18Z",
  "source_url": "https://dev.to/zeromathai/how-transformer-decoders-generate-text-from-causal-masking-to-decoding-1fh8",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 9221,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/9acbf4f536e7/pages/source.html",
  "local_html": "attachments/9acbf4f536e7/pages/article.html",
  "body_base_path": "attachments/9acbf4f536e7/pages/",
  "body_markdown_path": "attachments/9acbf4f536e7/pages/article.md"
}