{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Wesam Khallaf",
  "title_listed": "PyTorch Broadcasting Explained: The 3 Rules (and the Silent Bug That Bites Everyone)",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/pytorchfromgroundup/pytorch-broadcasting-explained-the-3-rules-and-the-silent-bug-that-bites-everyone-3606",
  "publish_date": "2026-07-16",
  "corpus_stage": "",
  "notes": "",
  "id": "2ef9b35b85dd",
  "collected_at": "2026-10-02T13:54:05",
  "title": "PyTorch Broadcasting Explained: The 3 Rules (and the Silent Bug That Bites Everyone) - DEV Community",
  "body_markdown": "You added two tensors of different shapes and PyTorch didn't complain — it just returned something bigger than you expected. Or you got a\nRuntimeError\nabout sizes that don't match and no idea which dimension is wrong. Both come down to one feature: broadcasting.\nThe short answer\nPyTorch broadcasting follows exactly three rules:\nalign shapes from the right, each aligned pair of dimensions must be equal or one of them must be 1, and any size-1 dimension is virtually stretched to match the other.\nIf a pair is neither equal nor 1, you get a\nRuntimeError\n. The stretch never copies memory — it's implemented with a stride of 0. Learn the three rules once and broadcasting stops surprising you.\nThe core example\nAdd a column of three numbers to a row of four:\nimport\ntorch\na\n=\ntorch\n.\ntensor\n([[\n1\n],\n[\n2\n],\n[\n3\n]])\n# shape (3, 1)\nb\n=\ntorch\n.\ntensor\n([[\n10\n,\n20\n,\n30\n,\n40\n]])\n# shape (1, 4)\nc\n=\na\n+\nb\nprint\n(\nc\n.\nshape\n)\nprint\n(\nc\n)\nEnter fullscreen mode\nExit fullscreen mode\nOutput:\ntorch\n.\nSize\n([\n3\n,\n4\n])\ntensor\n([[\n11\n,\n21\n,\n31\n,\n41\n],\n[\n12\n,\n22\n,\n32\n,\n42\n],\n[\n13\n,\n23\n,\n33\n,\n43\n]])\nEnter fullscreen mode\nExit fullscreen mode\nNo loop, no manual copying. PyTorch stretched the column across four columns and the row down three rows, then added element by element. Critically, the stretch is\nvirtual\n— no memory is allocated for the repeated values, which is why broadcasting is fast.\nThe 3 broadcasting rules\nAlign from the right.\nLine the shapes up starting at the last dimension. If one tensor has fewer dimensions, pad it with 1s on the left.\nEach pair must match.\nFor each aligned pair, the sizes must be equal, or one of them must be 1.\nSize-1 dims stretch.\nAny dimension of size 1 is virtually repeated to match the other tensor along that axis.\nWalking the core example through the rules:\ndim0   dim1\na         3      1   ->  stretches to 4\nb         1      4   ->  stretches to 3   (b padded to (1, 4))\nresult    3      4\nEnter fullscreen mode\nExit fullscreen mode\nThe output shape takes the max of each aligned pair:\n(3, 4)\n.\nWorked examples\nScalar + matrix.\nA scalar has shape\n()\nand pads to match anything:\nm\n=\ntorch\n.\nones\n(\n3\n,\n4\n)\nprint\n((\nm\n+\n10\n).\nshape\n)\n# (3, 4)\nEnter fullscreen mode\nExit fullscreen mode\nVector + matrix.\nA\n(4,)\nvector aligns with the last dim of a\n(3, 4)\nmatrix — it's implicitly treated as\n(1, 4)\n, then stretched down three rows:\nv\n=\ntorch\n.\ntensor\n([\n1.0\n,\n2.0\n,\n3.0\n,\n4.0\n])\n# (4,)\nm\n=\ntorch\n.\nones\n(\n3\n,\n4\n)\n# (3, 4)\nprint\n((\nm\n+\nv\n).\nshape\n)\n# (3, 4)\nEnter fullscreen mode\nExit fullscreen mode\nPer-channel image normalization.\nSubtract a\n(3, 1, 1)\nmean from a\n(3, 224, 224)\nimage — this is standard ImageNet normalization, and the size-1 dims are exactly what broadcasting knows how to stretch:\nimg\n=\ntorch\n.\nrandn\n(\n3\n,\n224\n,\n224\n)\nmean\n=\ntorch\n.\ntensor\n([.\n485\n,\n.\n456\n,\n.\n406\n])\nmean\n=\nmean\n[:,\nNone\n,\nNone\n]\n# (3, 1, 1)\nout\n=\nimg\n-\nmean\n# (3, 224, 224)\nprint\n(\nout\n.\nshape\n)\nEnter fullscreen mode\nExit fullscreen mode\nWhy does broadcasting fail with a RuntimeError?\nIf a pair of aligned dimensions is neither equal nor 1, PyTorch raises a\nRuntimeError\n— it's protecting you from a silent mistake:\na\n=\ntorch\n.\nones\n(\n3\n)\nb\n=\ntorch\n.\nones\n(\n5\n)\na\n+\nb\n# RuntimeError!\nEnter fullscreen mode\nExit fullscreen mode\nRuntimeError: The size of tensor a (3) must match the size of tensor b (5)\nat non-singleton dimension 0\nEnter fullscreen mode\nExit fullscreen mode\nThe fix is always the same: reshape one tensor so the mismatched dimension becomes size 1 (with\nunsqueeze\nor\nreshape\n), then let broadcasting do the rest. If the shapes genuinely shouldn't combine, the error just saved you from a bug.\nThe silent broadcasting bug (this one has no error)\nThe real danger isn't the\nRuntimeError\n— that message tells you exactly what's wrong. The danger is when broadcasting\nsucceeds silently\nand hands you a bigger tensor than you meant:\na\n=\ntorch\n.\nrandn\n(\n3\n)\n# (3,)\nb\n=\ntorch\n.\nrandn\n(\n3\n,\n1\n)\n# (3, 1)\nc\n=\na\n-\nb\nprint\n(\nc\n.\nshape\n)\n# (3, 3)  <-- probably wrong!\nEnter fullscreen mode\nExit fullscreen mode\nYou likely wanted an element-wise\n(3,)\nresult. Instead\n(3,)\nwas padded to\n(1, 3)\n,\n(3, 1)\nstayed as is, and you got a 3×3 matrix. No error, no warning — just quietly wrong, and it compounds ten lines later. This is the number-one broadcasting bug.\nBroadcasting and memory: the stride-0 trick\nThe stretch is virtual. Internally PyTorch uses a\nstride of 0\nalong the stretched dimension, so the same single row or column is reused for every position:\na\n=\ntorch\n.\nones\n(\n1\n,\n4\n)\ne\n=\na\n.\nexpand\n(\n3\n,\n4\n)\n# explicit broadcast\nprint\n(\ne\n.\nshape\n)\n# (3, 4)\nprint\n(\ne\n.\nstride\n())\n# (0, 1)  <- stride 0!\nprint\n(\ne\n.\nnumel\n())\n# 12 \"virtual\" elements\nprint\n(\na\n.\nnumel\n())\n# 4 actual elements\nEnter fullscreen mode\nExit fullscreen mode\nThe expanded tensor reports 12 elements but only 4 exist in memory. Stride 0 means \"don't advance — reuse the same row.\" That's why broadcasting is fast even when the virtual expanded tensor would be huge.\nCommon mistakes and gotchas\nSubtracting\n(3,)\nand\n(3, 1)\nexpecting\n(3,)\n.\nYou get\n(3, 3)\n. Make both shapes match exactly for element-wise.\nMismatched non-1 dimensions.\n(3,) + (5,)\nraises a\nRuntimeError\nat the non-singleton dimension.\nNot knowing which dim stretched.\nPrint\n.shape\nbefore\nand\nafter every broadcast until the habit is automatic.\nForgetting to add size-1 dims.\nPer-channel ops need an explicit\nunsqueeze\n/\nNone\nto line up the axes.\nA three-line habit that catches every broadcasting bug\n# 1. Print both shapes\nprint\n(\na\n.\nshape\n,\nb\n.\nshape\n)\n# 2. Align from the right on paper:  a: 3,1   b: 1,4  -> out 3,4 — is that what you want?\n# 3. If not, unsqueeze or reshape to control which dim stretches\nEnter fullscreen mode\nExit fullscreen mode\nPrint, align, decide. That habit catches broadcasting bugs before they turn into a wrong result downstream.\nRecap\nAlign from the right, each pair must be equal or one must be 1, and size-1 dims stretch for free. The output takes the max of each pair. A mismatch raises a\nRuntimeError\n; the sneakier failure is a silent blow-up that returns the wrong shape — so print your shapes.\nThis is one idea from my book\nPyTorch From Ground Up\n, which builds everything from tensors upward so nothing stays vague. If it helped, you can grab the\nfree PyTorch tensor cheat-sheet here\n, run every example from the book in the\ncompanion notebooks on GitHub\n, or find the\nfull paperback on Amazon here\n.",
  "metadata": {
    "title": "PyTorch Broadcasting Explained: The 3 Rules (and the Silent Bug That Bites Everyone) - DEV Community",
    "description": "You added two tensors of different shapes and PyTorch didn't complain — it just returned something... Tagged with python, pytorch, broadcasting, machinelearning.",
    "keywords": "python, pytorch, broadcasting, machinelearning, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/pytorchfromgroundup/pytorch-broadcasting-explained-the-3-rules-and-the-silent-bug-that-bites-everyone-3606",
    "og:title": "PyTorch Broadcasting Explained: The 3 Rules (and the Silent Bug That Bites Everyone)",
    "og:description": "You added two tensors of different shapes and PyTorch didn't complain — it just returned something...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdy98ug2cfafcacvigioo.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@wesamkhallaf",
    "twitter:title": "PyTorch Broadcasting Explained: The 3 Rules (and the Silent Bug That Bites Everyone)",
    "twitter:description": "You added two tensors of different shapes and PyTorch didn't complain — it just returned something...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdy98ug2cfafcacvigioo.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/2ef9b35b85dd/code/body-000.txt",
      "chars": 151,
      "language": "text"
    },
    {
      "file": "attachments/2ef9b35b85dd/code/body-001.txt",
      "chars": 98,
      "language": "text"
    },
    {
      "file": "attachments/2ef9b35b85dd/code/body-002.txt",
      "chars": 142,
      "language": "text"
    },
    {
      "file": "attachments/2ef9b35b85dd/code/body-003.txt",
      "chars": 54,
      "language": "text"
    },
    {
      "file": "attachments/2ef9b35b85dd/code/body-004.txt",
      "chars": 131,
      "language": "text"
    },
    {
      "file": "attachments/2ef9b35b85dd/code/body-005.txt",
      "chars": 174,
      "language": "text"
    },
    {
      "file": "attachments/2ef9b35b85dd/code/body-006.txt",
      "chars": 60,
      "language": "text"
    },
    {
      "file": "attachments/2ef9b35b85dd/code/body-007.txt",
      "chars": 104,
      "language": "text"
    },
    {
      "file": "attachments/2ef9b35b85dd/code/body-008.txt",
      "chars": 128,
      "language": "text"
    },
    {
      "file": "attachments/2ef9b35b85dd/code/body-009.txt",
      "chars": 233,
      "language": "text"
    },
    {
      "file": "attachments/2ef9b35b85dd/code/body-010.txt",
      "chars": 202,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://payhip.com/b/7ukxh",
      "label": "free PyTorch tensor cheat-sheet here",
      "type": "a"
    },
    {
      "href": "https://github.com/pytorch-from-ground-up/book_code",
      "label": "companion notebooks on GitHub",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/dp/B0H8WMCV33",
      "label": "full paperback on Amazon here",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:54Z",
  "source_url": "https://dev.to/pytorchfromgroundup/pytorch-broadcasting-explained-the-3-rules-and-the-silent-bug-that-bites-everyone-3606",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 6419,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/2ef9b35b85dd/pages/source.html",
  "local_html": "attachments/2ef9b35b85dd/pages/article.html",
  "body_base_path": "attachments/2ef9b35b85dd/pages/",
  "body_markdown_path": "attachments/2ef9b35b85dd/pages/article.md"
}