{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "zeromathai",
  "title_listed": "How Transformers Work — From Self-Attention to Modern LLM Architecture",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/zeromathai/how-transformers-work-from-self-attention-to-modern-llm-architecture-4j1o",
  "publish_date": "2026-06-15",
  "corpus_stage": "",
  "notes": "",
  "id": "0a0cfca28bef",
  "collected_at": "2026-10-02T13:50:48",
  "title": "How Transformers Work — From Self-Attention to Modern LLM Architecture - DEV Community",
  "body_markdown": "Transformers changed AI because they stopped reading sequences one token at a time.\nInstead of moving step by step like an RNN, a Transformer compares tokens directly.\nThat one design shift made modern LLMs possible.\nCore Idea\nA Transformer is a neural network architecture built around attention.\nIt looks at a sequence of tokens and learns how those tokens relate to each other.\nThis matters because language is contextual.\nA word is not understood alone.\nIt is understood through its relationship with surrounding words.\nThat is why Self-Attention became the core mechanism.\nThe Key Structure\nA simplified Transformer flow looks like this:\nTokens → Embeddings → Positional Information → Self-Attention → Feed-Forward Network → Output\nMore compactly:\nTransformer = token representations + attention + position + stacked blocks\nThe model first converts text into token vectors.\nThen it injects position information.\nThen each Transformer block updates the token representations using attention and feed-forward layers.\nImplementation View\nAt a high level, a Transformer processes text like this:\nsplit text into tokens\n\nconvert tokens into embeddings\n\nadd positional information\n\nfor each Transformer block:\n    compute Self-Attention\n\n    mix token information\n\n    apply feed-forward transformation\n\n    keep stable flow with residual connections and normalization\n\nproduce contextual token representations\nEnter fullscreen mode\nExit fullscreen mode\nFor decoder-based LLMs, generation continues like this:\npredict next token\n\nappend generated token\n\nreuse cached keys and values\n\nrepeat until stopping condition\nEnter fullscreen mode\nExit fullscreen mode\nThis is why Transformers are practical for large-scale generation.\nThey can learn relationships across many tokens.\nAnd with caching, they can generate efficiently.\nConcrete Example\nTake this sentence:\nThe animal did not cross the street because it was tired.\nWhat does “it” refer to?\nA simple left-to-right model may struggle if long context matters.\nSelf-Attention lets the token “it” compare itself with other tokens like “animal” and “street.”\nThe model can assign stronger attention to the token that best explains the meaning.\nThat is the intuition.\nAttention lets tokens ask:\nWhich other tokens matter for understanding me?\nRNN vs Transformer\nThis comparison explains why Transformers became so important.\nRNN:\nprocesses tokens step by step\ncarries information through hidden state\nnaturally captures order\nis harder to parallelize\ncan struggle with long-range dependencies\nTransformer:\nprocesses tokens in parallel\ncompares tokens directly through attention\nneeds positional information for order\nscales well on GPUs\nhandles long-range relationships more flexibly\nSo the Transformer was not just faster.\nIt changed how sequence relationships are represented.\nRNNs remember through recurrence.\nTransformers relate through attention.\nSelf-Attention\nSelf-Attention computes relationships between tokens in the same sequence.\nEach token creates three vectors:\nQuery\nKey\nValue\nThe intuition is simple:\nQuery = what this token is looking for\nKey = what each token offers for matching\nValue = information to retrieve if the match is strong\nThe core formula is:\nAttention(Q, K, V) = softmax((QK^T) / sqrt(d_k))V\nThis means:\ncompare queries and keys\nturn scores into weights\nuse those weights to combine values\nThat is how each token becomes context-aware.\nMulti-Head Attention\nOne attention calculation is useful.\nBut one view is not enough.\nMulti-Head Attention runs several attention heads in parallel.\nEach head can focus on a different type of relationship.\nOne head may track syntax.\nAnother may track semantic similarity.\nAnother may track long-distance references.\nThen the outputs are combined into one representation.\nThis makes attention richer than a single similarity calculation.\nWhy Positional Encoding Is Needed\nSelf-Attention does not automatically know token order.\nIf you only give it a bag of token embeddings, the model needs another signal to know which token came first.\nThat is why positional information is added.\nCommon positional methods include:\nAbsolute Positional Embedding\nRelative Positional Embedding\nRotary Positional Embedding\nAPE gives each position its own vector.\nRPE focuses on relative distance between tokens.\nRoPE rotates query and key vectors based on position, making relative position work naturally inside attention.\nThis is why RoPE became common in modern LLMs.\nEncoder, Decoder, and LLMs\nThe original Transformer used an Encoder-Decoder structure.\nEncoder:\nreads the input\nbuilds contextual representations\nworks well for understanding tasks\nDecoder:\ngenerates output tokens\nuses causal masking\nworks well for autoregressive generation\nEncoder-Decoder:\nconnects input understanding with output generation\nuseful for translation-style tasks\nModern GPT-style LLMs are mostly decoder-based.\nThey generate text one token at a time.\nThe decoder predicts the next token, appends it, and repeats.\nDecoding Strategies\nOnce the model produces logits, it needs to choose the next token.\nDifferent decoding strategies create different behavior.\nGreedy decoding:\nchooses the most likely token\nsimple and deterministic\ncan be repetitive\nBeam search:\nkeeps multiple candidate sequences\nuseful for structured generation\ncan still feel less diverse\nTop-k sampling:\nsamples from the top k likely tokens\nadds diversity\nTop-p sampling:\nsamples from the smallest probability mass above a threshold\nadapts the candidate set dynamically\nSo generation quality is not only about the model.\nIt also depends on decoding.\nThe Efficiency Problem\nFull Attention is powerful but expensive.\nIf the sequence length is n, attention has roughly O(n^2) cost.\nThat means longer context becomes expensive quickly.\nThis is why efficient attention matters.\nLocal Attention reduces the view to nearby tokens.\nSparse Attention computes only selected attention links.\nFlashAttention keeps the formula but improves GPU memory access.\nThe key idea:\nDo less unnecessary work, or move data more efficiently.\nBoth make longer context more practical.\nKV Cache\nAutoregressive generation has another problem.\nWhen generating one token at a time, the model repeatedly needs past key and value tensors.\nKV Cache stores those tensors.\nSo the model does not recompute them from scratch at every step.\nThe flow looks like this:\nGenerated tokens → cached keys and values → new query attends to cache → next token\nThis makes inference faster.\nBut it creates a memory problem.\nLonger context means a larger KV Cache.\nThat is why modern LLMs use techniques like:\nMulti-Query Attention\nGrouped-Query Attention\nMulti-Head Latent Attention\nThese methods reduce the memory cost of storing key-value information.\nModern Transformer Blocks\nModern LLMs still use the Transformer idea.\nBut the block has evolved.\nA typical modern block looks like this:\nInput\n→ RMSNorm or Pre-Layer Normalization\n→ Self-Attention with GQA and RoPE\n→ Residual Connection\n→ RMSNorm or Pre-Layer Normalization\n→ Feed-Forward Network with SwiGLU or Mixture of Experts\n→ Residual Connection\nImportant upgrades include:\nRMSNorm for simpler normalization\nRoPE for positional representation\nGQA for efficient inference\nSwiGLU for stronger feed-forward layers\nMoE for sparse expert-based scaling\nSo today’s Transformer is not exactly the 2017 Transformer copied directly.\nIt is an evolved architecture family.\nTransformer vs Modern LLM Architecture\nOriginal Transformer:\nencoder-decoder structure\nstandard multi-head attention\nsinusoidal positional encoding\nlayer normalization\ndense feed-forward layers\nModern LLM architecture:\noften decoder-only\ncausal self-attention\nRoPE\nRMSNorm\nGQA or related KV-sharing methods\nSwiGLU\nsometimes Mixture of Experts\nKV Cache for inference\nThe core idea stayed the same.\nThe engineering changed dramatically.\nRecommended Learning Order\nIf Transformer architecture feels too large, learn it in this order:\nAttention Mechanism\nSelf-Attention\nQKV Computation\nMulti-Head Attention\nPositional Encoding\nEncoder-Decoder Architecture\nTransformer Decoder\nKV Cache\nEfficient Attention\nModern Transformer Block\nThis order works because you first understand the relationship mechanism.\nThen you understand generation.\nThen you understand why modern LLMs needed efficiency upgrades.\nTakeaway\nThe Transformer is the architecture language of modern LLMs.\nThe shortest version is:\nTransformer = attention + position + stacked blocks + efficient generation\nSelf-Attention computes token relationships.\nPositional encoding injects order.\nThe decoder generates tokens.\nKV Cache makes autoregressive inference practical.\nModern upgrades like RoPE, RMSNorm, GQA, SwiGLU, and MoE make the architecture scalable.\nIf you remember one idea, remember this:\nTransformers work by turning a sequence into a set of contextual relationships, then refining those relationships through stacked attention-based blocks.\nDiscussion\nWhen learning Transformers, do you find it easier to start from the attention formula, the decoder generation loop, or the modern LLM block structure?\nOriginally published at zeromathai.com.\nOriginal article:\nhttps://zeromathai.com/en/transformer-architecture-overview-en/\nGitHub Resources\nAI diagrams, study notes, and visual guides:\nhttps://github.com/zeromathai/zeromathai-ai",
  "metadata": {
    "title": "How Transformers Work — From Self-Attention to Modern LLM Architecture - DEV Community",
    "description": "Transformers changed AI because they stopped reading sequences one token at a time.  Instead of... Tagged with ai, machinelearning, llm, deeplearning.",
    "keywords": "ai, machinelearning, llm, deeplearning, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/zeromathai/how-transformers-work-from-self-attention-to-modern-llm-architecture-4j1o",
    "og:title": "How Transformers Work — From Self-Attention to Modern LLM Architecture",
    "og:description": "Transformers changed AI because they stopped reading sequences one token at a time.  Instead of...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F6grxhyaka3yk6ua2kw4v.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "How Transformers Work — From Self-Attention to Modern LLM Architecture",
    "twitter:description": "Transformers changed AI because they stopped reading sequences one token at a time.  Instead of...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F6grxhyaka3yk6ua2kw4v.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/0a0cfca28bef/code/body-000.txt",
      "chars": 313,
      "language": "text"
    },
    {
      "file": "attachments/0a0cfca28bef/code/body-001.txt",
      "chars": 106,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://zeromathai.com/en/transformer-architecture-overview-en/",
      "label": "https://zeromathai.com/en/transformer-architecture-overview-en/",
      "type": "a"
    },
    {
      "href": "https://github.com/zeromathai/zeromathai-ai",
      "label": "https://github.com/zeromathai/zeromathai-ai",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:19Z",
  "source_url": "https://dev.to/zeromathai/how-transformers-work-from-self-attention-to-modern-llm-architecture-4j1o",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 9323,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/0a0cfca28bef/pages/source.html",
  "local_html": "attachments/0a0cfca28bef/pages/article.html",
  "body_base_path": "attachments/0a0cfca28bef/pages/",
  "body_markdown_path": "attachments/0a0cfca28bef/pages/article.md"
}