{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Priyesh Dave",
  "title_listed": "Day 27: Self-Attention Explained From Scratch",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/priyeshdave6/day-27-self-attention-explained-from-scratch-21j7",
  "publish_date": "2026-09-27",
  "corpus_stage": "",
  "notes": "",
  "id": "d9fdf982e7c9",
  "collected_at": "2026-10-02T13:49:01",
  "title": "Day 27: Self-Attention Explained From Scratch - DEV Community",
  "body_markdown": "What Problem Does Self-Attention Solve?\nUnderstanding sequences, like sentences, depends on how words relate to each other. In language, a word's meaning can depend on words that come much earlier or later in the sentence.\nTraditional sequence models, such as RNNs (Recurrent Neural Networks), process sequences one word at a time. Each step updates a memory that carries information forward. Passing information from the start of a sentence to the end requires many steps, which makes it easy to lose details—especially in long sentences.\nCNNs (Convolutional Neural Networks) look at groups of nearby words to spot patterns. But connecting information between distant words, like the beginning and end of a sentence, takes stacking many CNN layers. This quickly becomes inefficient.\nBoth of these approaches struggle when long-range relationships matter. For instance, in the sentence “The animal, which the farmer chased, ran away,” the subject “animal” pairs with the verb “ran away,” but they're separated by several words. We need a model that gives every word direct access to all others, so it can use whatever context matters, even across long distances.\nSelf-attention solves this problem. It lets every word see and weigh all the other words in the sequence simultaneously.\nWhat Is Self-Attention (in Plain Terms)?\nSelf-attention is a process where each word looks at all other words in the sequence to decide which are important for its own representation. Instead of marching step by step, every word gets information from everywhere else, all at once.\nThink of reading a sentence. For each word, you glance at the rest of the sentence to check what changes its meaning. Some words might barely matter, others might be essential.\nAnalogy: Imagine a group meeting. Each person (word) listens to everyone else before forming their opinion. You might care a lot about what one person says, and barely note another. The weight you give to each speaker shapes your final thought.\nThe Ingredients of Self-Attention: Queries, Keys, and Values\nSelf-attention uses three parts for every input word: a\nquery\n, a\nkey\n, and a\nvalue\n. These are just different views of the same word, computed as simple linear projections.\nQuery:\nWhat this word wants to find out about its context. It asks, “Which other words could change my meaning?”\nKey:\nWhat this word offers as information. It describes each word's own characteristics.\nValue:\nThe actual content this word provides, if chosen as relevant.\nLet’s use an example:\n\"The hungry cat chased the mouse.\"\nWhen focusing on “cat”:\nThe query for “cat” is like asking, “What do I need to know from this sentence?”\nEach other word, including “cat” itself, has a key (“I’m a state word,” “I’m an animal,” “I’m an action,” etc.) and a value (the information it holds).\n“Cat\" will attend strongly to “hungry” (its state) and “chased” (its action), based on how well its query matches their keys.\n“Chased\" will attend to “cat” and “mouse.”\nThe query, key, and value for each word are just different mathematical transformations (matrix multiplications) of its word embedding—the basic numeric representation of a word.\nHow Self-Attention Works Step by Step\nLet's walk through a concrete example using tiny vectors, so you can see every math step.\nSuppose our vocabulary consists of just three words: A, B, and C. Each word is represented as a 2-component vector.\nLet’s define:\nA = [1, 0]\nB = [0, 1]\nC = [1, 1]\nOur sequence is [A, B, C].\nTo keep things simple, we’ll set the query, key, and value for each word equal to the input vector. (Normally, you’d multiply by separate matrices. Here, we use the identity matrix, meaning nothing changes.)\nStep 1: Create Query, Key, and Value Matrices\nQ\n=\n[[\n1\n,\n0\n],\n# A’s query\n[\n0\n,\n1\n],\n# B’s query\n[\n1\n,\n1\n]]\n# C’s query\nK\n=\n[[\n1\n,\n0\n],\n# A’s key\n[\n0\n,\n1\n],\n# B’s key\n[\n1\n,\n1\n]]\n# C’s key\nV\n=\n[[\n1\n,\n0\n],\n# A’s value\n[\n0\n,\n1\n],\n# B’s value\n[\n1\n,\n1\n]]\n# C’s value\nEnter fullscreen mode\nExit fullscreen mode\nStep 2: Compute Attention Scores\nFor each word, calculate the dot product between its query and the keys of every word. This gives a score for how much each word might matter.\nFor A:\nA’s query · A’s key: [1,0]·[1,0] = 1\nA’s query · B’s key: [1,0]·[0,1] = 0\nA’s query · C’s key: [1,0]·[1,1] = 1\nA’s attention scores: [1, 0, 1]\nRepeat for B and C:\nB: [0,1]·[1,0]=0, [0,1]·[0,1]=1, [0,1]·[1,1]=1  → [0, 1, 1]\nC: [1,1]·[1,0]=1, [1,1]·[0,1]=1, [1,1]·[1,1]=2  → [1, 1, 2]\nStep 3: Normalize Scores with Softmax\nSoftmax is a math function that converts scores to positive numbers that sum to 1. This turns the raw scores into attention weights.\nFor A: softmax([1, 0, 1])\nFirst, exponentiate: exp(1)=2.718, exp(0)=1\nSum = 2.718 + 1 + 2.718 = 6.436\nWeights:\n[2.718/6.436, 1/6.436, 2.718/6.436] = [0.42, 0.16, 0.42]\nSo A pays 42% attention to itself, 16% to B, 42% to C.\nStep 4: Weighted Sum of Values\nEach word calculates its new representation as a weighted sum of all value vectors (one for each word), using its attention weights.\nFor A:\nOutput = 0.42 × [1, 0]  +  0.16 × [0, 1]  +  0.42 × [1, 1]\n= [0.42+0.42, 0.16+0.42]\n= [0.84, 0.58]\nRepeat this for B and C to get their new context-aware representations.\nNow, every output vector contains a blend of information from the whole sequence, mixed according to what each word considers important.\nMinimal Numpy Example\nimport\nnumpy\nas\nnp\n# Step 1: inputs (3 words, each as 2D vector)\nX\n=\nnp\n.\narray\n([\n[\n1.\n,\n0.\n],\n# Word A\n[\n0.\n,\n1.\n],\n# Word B\n[\n1.\n,\n1.\n]\n# Word C\n])\n# Step 2: queries, keys, values (use identity - no change)\nQ\n=\nX\nK\n=\nX\nV\n=\nX\n# Step 3: attention scores (dot product Q and K.T)\nscores\n=\nQ\n@\nK\n.\nT\n# Step 4: softmax (row-wise)\ndef\nsoftmax\n(\nx\n):\ne_x\n=\nnp\n.\nexp\n(\nx\n-\nnp\n.\nmax\n(\nx\n,\naxis\n=-\n1\n,\nkeepdims\n=\nTrue\n))\nreturn\ne_x\n/\ne_x\n.\nsum\n(\naxis\n=-\n1\n,\nkeepdims\n=\nTrue\n)\nweights\n=\nsoftmax\n(\nscores\n)\n# Step 5: attention output (weights @ V)\noutput\n=\nweights\n@\nV\nprint\n(\n\"\nAttention output vectors for A, B, C:\n\"\n)\nprint\n(\noutput\n)\nEnter fullscreen mode\nExit fullscreen mode\nThis code prints the new, context-enriched vector for each word after self-attention. Each result blends information from every word and shows how the output shifts based on what each word attended to.\nWhy Self-Attention Is Powerful\nSelf-attention lets every word immediately access and weigh all other words in the sequence. It doesn’t need to pass memory along step by step, like RNNs. It doesn’t have to rely on local windows, like CNNs.\nThis has big effects:\nParallelization:\nAll word-to-word calculations can run at once, making training fast with modern GPUs and TPUs.\nLong-Range Context:\nWords at the start and end of a sequence can influence each other directly, in a single operation.\nTask-Specific Weights:\nFor each word, the model learns which other words matter most for what it needs to do. There’s no fixed rule or window size.\nThese qualities made self-attention—especially as used in the Transformer model—a foundation for modern language and sequence models. It enables models to capture relationships in data more flexibly and effectively than previous approaches.\nKey Takeaways\nTraditional RNNs and CNNs struggle to connect distant words in sequences.\nSelf-attention allows every word to directly consider all others in a sentence or sequence.\nSelf-attention uses queries, keys, and values derived from word embeddings.\nThe mechanism involves matching queries to keys, normalizing with softmax, and combining values accordingly.\nSelf-attention enables efficient parallelization and captures long-range dependencies critical for models like Transformers.\nTry It Yourself\nTry implementing a simple self-attention calculation for three word embeddings: [1,0], [0,1], and [1,1]. Assign each as the query, key, and value for the corresponding word. For each word, compute dot products with all keys to get similarity scores, apply softmax to these scores, and then use the resulting weights to take a weighted average of the value vectors. Do this by hand or in code, and compare the computed outputs for each word.\nFurther Resources\n🎥\nAttention in transformers, step‑by‑step | 3Blue1Brown Deep Learning Chapter 6\n📄\nSelf‑Attention from Scratch — Tai Bui\n📄\nSelf‑attention from scratch (The Loss Curve)\n📄\nTransformers from Scratch | Sovesh Mohapatra\nComing up on Day 28:\nMulti-Head Attention",
  "metadata": {
    "title": "Day 27: Self-Attention Explained From Scratch - DEV Community",
    "description": "What Problem Does Self-Attention Solve?   Understanding sequences, like sentences, depends... Tagged with ai, nlp, genai, 100daysofcode.",
    "keywords": "ai, nlp, genai, 100daysofcode, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/priyeshdave6/day-27-self-attention-explained-from-scratch-21j7",
    "og:title": "Day 27: Self-Attention Explained From Scratch",
    "og:description": "What Problem Does Self-Attention Solve?   Understanding sequences, like sentences, depends...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnig4781tr69jgxrnz9iw.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Day 27: Self-Attention Explained From Scratch",
    "twitter:description": "What Problem Does Self-Attention Solve?   Understanding sequences, like sentences, depends...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnig4781tr69jgxrnz9iw.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/d9fdf982e7c9/code/body-000.txt",
      "chars": 239,
      "language": "text"
    },
    {
      "file": "attachments/d9fdf982e7c9/code/body-001.txt",
      "chars": 606,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=eMlx5fFNoYc",
      "status": "ok",
      "local_path": "attachments/d9fdf982e7c9/videos/eMlx5fFNoYc.mp4",
      "bytes": 151461784,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "1569.121000"
    }
  ],
  "links": [
    {
      "href": "https://www.youtube.com/watch?v=eMlx5fFNoYc",
      "label": "Attention in transformers, step‑by‑step | 3Blue1Brown Deep Learning Chapter 6",
      "type": "a"
    },
    {
      "href": "https://taibui.dev/phases/07-transformers-deep-dive/02-self-attention-from-scratch",
      "label": "Self‑Attention from Scratch — Tai Bui",
      "type": "a"
    },
    {
      "href": "https://thelosscurve.com/guides/attention-from-scratch",
      "label": "Self‑attention from scratch (The Loss Curve)",
      "type": "a"
    },
    {
      "href": "https://soveshmohapatra.com/projects/transformers/",
      "label": "Transformers from Scratch | Sovesh Mohapatra",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:02Z",
  "source_url": "https://dev.to/priyeshdave6/day-27-self-attention-explained-from-scratch-21j7",
  "papers": [],
  "actual_video_files": [
    {
      "local_path": "attachments/d9fdf982e7c9/videos/eMlx5fFNoYc.mp4",
      "bytes": 151461784,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "1569.121000",
      "status": "ok"
    }
  ],
  "body_chars": 8321,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/d9fdf982e7c9/pages/source.html",
  "local_html": "attachments/d9fdf982e7c9/pages/article.html",
  "body_base_path": "attachments/d9fdf982e7c9/pages/",
  "body_markdown_path": "attachments/d9fdf982e7c9/pages/article.md"
}