{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "zeromathai",
  "title_listed": "How Attention Actually Works — From Next-Token Prediction to QKV Intuition",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/zeromathai/how-attention-actually-works-from-next-token-prediction-to-qkv-intuition-29l2",
  "publish_date": "2026-06-17",
  "corpus_stage": "",
  "notes": "",
  "id": "9321decba019",
  "collected_at": "2026-10-02T13:50:46",
  "title": "How Attention Actually Works — From Next-Token Prediction to QKV Intuition - DEV Community",
  "body_markdown": "A language model does not “write sentences.”\nIt predicts the next token. One step at a time.\nSo the real question is:\nHow does it decide what matters right now?\nThat is why attention exists.\nCore Idea\nA Language Model = next-token probability estimator.\nGiven previous tokens, it predicts the next token.\nAttention = mechanism that decides which past tokens matter more.\nThis is critical.\nBecause not all context is equally useful.\nThe Key Structure\nLanguage Modeling can be reduced to:\nP(x₁, x₂, ..., xₜ) = Π P(xₜ | x₁...xₜ₋₁)\nAnd attention adds:\nweighted context selection\nMore concretely:\nLanguage Model = context + weighting + prediction\nWithout attention:\nAll context is compressed.\nWith attention:\nContext is dynamically re-weighted at every step.\nPseudo-code View\nAutoregressive generation:\ncontext = [\"I\", \"love\"]\n\nwhile not finished:\n    probs = model(context)\n    next_token = sample(probs)\n\n    context.append(next_token)\nEnter fullscreen mode\nExit fullscreen mode\nAttention inside the model:\nfor each token t:\n    score = compare(query_t, keys)\n\n    weights = softmax(score)\n\n    output_t = sum(weights * values)\nEnter fullscreen mode\nExit fullscreen mode\nThat is the core loop.\nPredict → append → repeat.\nConcrete Example\nInput:\n\"I love\"\nPossible next tokens:\nyou, it, this, pizza\nThe model assigns probabilities:\nyou → 0.6\nit → 0.2\nthis → 0.1\npizza → 0.1\nWhy does “you” win?\nBecause attention focuses on relationships in context.\n“I” + “love” → strong pattern → “you”\nNow extend:\n\"I love you because\"\nThe model must now decide:\nWhat does “because” relate to?\nAttention allows it to re-evaluate the entire context.\nNot just the last token.\nWhy Attention Is Needed\nOld Seq2Seq models:\ncompress entire input into one vector\nlose information as sequence grows\nAttention fixes this:\nkeeps all token representations\ndynamically selects relevant ones\nThis matters because:\nLong sentences break fixed representations.\nAttention removes that bottleneck.\nQKV Intuition\nAttention uses three vectors:\nQuery, Key, Value\nThink like search:\nQuery = what I want\nKey = what each token offers\nValue = the actual information\nFlow:\ncompare Query with Keys\ncompute similarity scores\nnormalize with softmax\ncombine Values using weights\nThat is how context is selected.\nThe Core Formula\nAttention is:\nAttention(Q, K, V) = softmax(QKᵀ / √d) V\nMeaning:\nmatch Query with Keys\nturn matches into probabilities\nuse those probabilities to mix Values\nResult:\nEach token becomes context-aware.\nCross Attention and Context Vector\nIn encoder-decoder models:\nDecoder does not rely only on its own tokens.\nIt looks at Encoder outputs.\nContext vector:\nc = Σ (attention_weight × encoder_hidden_state)\nThis is dynamic.\nAt every step, the model recomputes what matters.\nNot a fixed summary.\nA moving focus.\nNaive vs Real View\nNaive view:\nLanguage model = next word generator\nReal view:\nLanguage model = dynamic context weighting system\nNaive:\npredict next token\nEnter fullscreen mode\nExit fullscreen mode\nReal:\ncompute attention\nreweight context\nthen predict token\nEnter fullscreen mode\nExit fullscreen mode\nThat difference is everything.\nIt explains why Transformers outperform older models.\nImportant Constraints\nAttention is powerful, but not free.\ncost grows with sequence length\nrequires memory for all tokens\ndepends on good tokenization\nstill generates sequentially at inference\nAlso:\nAttention does not understand meaning by itself.\nIt only learns patterns from data.\nSo quality depends on training.\nWhy This Matters (Again)\nEarly:\nWithout attention → information bottleneck\nNow:\nWith attention → full context + selective focus\nThis is why modern LLMs work.\nNot because they “know language.”\nBut because they efficiently manage context.\nTakeaway\nLanguage Model = next-token prediction.\nAttention = context selection.\nQKV = mechanism for selecting information.\nIf you remember one thing:\nAttention lets a model decide what to look at before predicting what to say.\nThat is the core of modern LLMs.\nDiscussion\nWhen you think about LLM behavior, do you see it more as:\na probability engine or a context selection system?\nOriginally published at zeromathai.com\nOriginal article:\nhttps://zeromathai.com/en/attention-language-modeling-basics-en/\nGitHub Resources\nAI diagrams, study notes, and visual guides:\nhttps://github.com/zeromathai/zeromathai-ai",
  "metadata": {
    "title": "How Attention Actually Works — From Next-Token Prediction to QKV Intuition - DEV Community",
    "description": "A language model does not “write sentences.”  It predicts the next token. One step at a time.  So the... Tagged with ai, machinelearning, nlp, transformers.",
    "keywords": "ai, machinelearning, nlp, transformers, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/zeromathai/how-attention-actually-works-from-next-token-prediction-to-qkv-intuition-29l2",
    "og:title": "How Attention Actually Works — From Next-Token Prediction to QKV Intuition",
    "og:description": "A language model does not “write sentences.”  It predicts the next token. One step at a time.  So the...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fb5m5f76nysx75pvibnt1.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "How Attention Actually Works — From Next-Token Prediction to QKV Intuition",
    "twitter:description": "A language model does not “write sentences.”  It predicts the next token. One step at a time.  So the...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fb5m5f76nysx75pvibnt1.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/9321decba019/code/body-000.txt",
      "chars": 135,
      "language": "text"
    },
    {
      "file": "attachments/9321decba019/code/body-001.txt",
      "chars": 121,
      "language": "text"
    },
    {
      "file": "attachments/9321decba019/code/body-002.txt",
      "chars": 19,
      "language": "text"
    },
    {
      "file": "attachments/9321decba019/code/body-003.txt",
      "chars": 54,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://zeromathai.com/en/attention-language-modeling-basics-en/",
      "label": "https://zeromathai.com/en/attention-language-modeling-basics-en/",
      "type": "a"
    },
    {
      "href": "https://github.com/zeromathai/zeromathai-ai",
      "label": "https://github.com/zeromathai/zeromathai-ai",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:19Z",
  "source_url": "https://dev.to/zeromathai/how-attention-actually-works-from-next-token-prediction-to-qkv-intuition-29l2",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 4327,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/9321decba019/pages/source.html",
  "local_html": "attachments/9321decba019/pages/article.html",
  "body_base_path": "attachments/9321decba019/pages/",
  "body_markdown_path": "attachments/9321decba019/pages/article.md"
}