{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "AI by Hand ✍️ Prof. Tom Yeh",
  "title_listed": "Attention Computation by Hand",
  "keywords": "注意力,矩阵计算,手算",
  "domain": "大模型算法",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://www.byhand.ai/p/library-models-attention-ktq-score",
  "publish_date": "2025-01",
  "corpus_stage": "预训练/SFT",
  "notes": "手算注意力计算",
  "id": "fb6f605f9083",
  "collected_at": "2026-10-01T20:45:43",
  "title": "Attention Computation - by Prof. Tom Yeh - AI by Hand ✍️",
  "body_markdown": "Library\n›\nAttention\nQKV Projection\nAttention Computation\nSelf Attention\nCross Attention\nSelf Attention vs Cross Attention\nSelf Attention (Shared KV)\nMulti-Head Attention\nFused QKV (Multi-Head)\nSingle vs Multi-Head Attention\nMulti-Query Attention\nGrouped-Query Attention\nYou have your query (\"who has a dog?\"), every neighbor's key, and every neighbor's address. Now you score each neighbor against your query.\nOpen the interactive diagram ↗\nS = Kᵀ × Q\ncomputes one score per (neighbor, querier) pair. The neighbor with a dog scores high. The neighbor with a cat scores near zero. S is a seq × seq matrix: every token scoring every other token simultaneously.\nThe scores are divided by √d (key dimension) before softmax. Without this, dot products grow large as the key dimension increases, and softmax collapses: you'd fixate on one neighbor and ignore everyone else.\nA = softmax(S/√d)\nconverts scores into attention weights. Each column of A sums to 1: your attention is a probability distribution over all neighbors. The dog owner gets most of the weight.\nF = V × A\ntakes a weighted combination of all addresses. The dog owner's address dominates the output. In practice this blend is the whole point: the model does not pick one token but mixes information from all of them, weighted by relevance.\nThis step has no learned parameters. It is just matrix multiplications and a softmax. It is also what makes self-attention O(n²): S and A are both seq × seq regardless of key or value dimensions.\nNext:\n3. Self Attention",
  "metadata": {
    "title": "Attention Computation - by Prof. Tom Yeh - AI by Hand ✍️",
    "description": "Attention: 2 of 11",
    "keywords": null,
    "author": "Prof. Tom Yeh",
    "og:type": "article",
    "og:title": "Attention Computation",
    "og:description": "Attention: 2 of 11",
    "og:image": "https://substackcdn.com/image/fetch/$s_!OS_4!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F64aa24b8-ee3d-40f4-96f4-51452b09a692_1456x924.jpeg",
    "og:url": "https://www.byhand.ai/p/library-models-attention-ktq-score",
    "twitter:title": "Attention Computation",
    "twitter:description": "Attention: 2 of 11",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!D6RI!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Faibyhand.substack.com%2Fapi%2Fv1%2Fpost_preview%2F199222367%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2026-06-29T19:15:23.150Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!OS_4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F64aa24b8-ee3d-40f4-96f4-51452b09a692_1456x924.jpeg",
      "bytes": 15457,
      "reused": true,
      "local_path": "attachments/fb6f605f9083/images/039.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!OS_4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F64aa24b8-ee3d-40f4-96f4-51452b09a692_1456x924.jpeg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.byhand.ai/p/library",
      "label": "Library",
      "type": "a"
    },
    {
      "href": "https://www.byhand.ai/p/library-series-models-attention",
      "label": "Attention",
      "type": "a"
    },
    {
      "href": "https://www.byhand.ai/p/library-models-attention-qkv-projection",
      "label": "QKV Projection",
      "type": "a"
    },
    {
      "href": "https://www.byhand.ai/p/library-models-attention-self-attention",
      "label": "Self Attention",
      "type": "a"
    },
    {
      "href": "https://www.byhand.ai/p/library-models-attention-cross-attention",
      "label": "Cross Attention",
      "type": "a"
    },
    {
      "href": "https://www.byhand.ai/p/library-models-attention-self-vs-cross",
      "label": "Self Attention vs Cross Attention",
      "type": "a"
    },
    {
      "href": "https://www.byhand.ai/p/library-models-attention-self-attention-shared-kv",
      "label": "Self Attention (Shared KV)",
      "type": "a"
    },
    {
      "href": "https://www.byhand.ai/p/library-models-attention-mha",
      "label": "Multi-Head Attention",
      "type": "a"
    },
    {
      "href": "https://www.byhand.ai/p/library-models-attention-fused-qkv",
      "label": "Fused QKV (Multi-Head)",
      "type": "a"
    },
    {
      "href": "https://www.byhand.ai/p/library-models-attention-single-vs-multi-head",
      "label": "Single vs Multi-Head Attention",
      "type": "a"
    },
    {
      "href": "https://www.byhand.ai/p/library-models-attention-multi-query-attention",
      "label": "Multi-Query Attention",
      "type": "a"
    },
    {
      "href": "https://www.byhand.ai/p/library-models-attention-gqa",
      "label": "Grouped-Query Attention",
      "type": "a"
    },
    {
      "href": "https://library.byhand.ai/embed/models/attention/ktq-score",
      "label": "Open the interactive diagram ↗",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T11:11:41Z",
  "source_url": "https://www.byhand.ai/p/library-models-attention-ktq-score",
  "delivery_status": "ok",
  "discovery_status": "ok",
  "missing_reason": "",
  "repair_error": "",
  "actual_video_files": [],
  "papers": [],
  "body_chars": 1520,
  "expected_body_images": 1,
  "raw_html": "attachments/fb6f605f9083/pages/source.html",
  "local_html": "attachments/fb6f605f9083/pages/article.html",
  "body_base_path": "attachments/fb6f605f9083/pages/",
  "body_markdown_path": "attachments/fb6f605f9083/pages/article.md"
}