{ "source_file": "0930", "source_platform": "Dev.to", "author": "Wesam Khallaf", "title_listed": "PyTorch Autograd Explained: What .backward() Actually Does", "keywords": "", "domain": "机器学习", "content_type": "技术教程", "is_paid_flag": "否", "url": "https://dev.to/pytorchfromgroundup/pytorch-autograd-explained-what-backward-actually-does-4acg", "publish_date": "2026-08-12", "corpus_stage": "", "notes": "", "id": "869bb105719b", "collected_at": "2026-10-02T13:53:57", "title": "PyTorch Autograd Explained: What .backward() Actually Does - DEV Community", "body_markdown": "most\nautograd\ntutorials show you\nrequires_grad\n, then .\nbackward\n(), then .\ngrad\n, and then they hand you a training loop, and you come out of it able to run autograd and still not able to say what it did, and i think that is backwards. so in this one you do the derivative first, by hand, on a tiny network that fits on one page, and only once you have your own number do we run pytorch and check it against .grad. the two agree to the decimal. and that is the point where autograd stops being magic, because you already did the thing it does, you just did it slower.\nFirst, what a gradient actually is\nSkip this section if you are comfortable with it, but most confusion about autograd is really confusion about what it is producing.\nImagine standing on a hillside in fog. You cannot see the bottom, but you can feel that the ground tilts. Step in the direction it tilts downward and you get lower. That is a gradient: the slope of the ground under your feet.\nNow put that on a graph. The horizontal axis is one adjustable number inside the model, a parameter. The vertical axis is the loss. The curve shows how the loss changes as the parameter\nchanges, and somewhere on it there is a lowest point.\nTake\ny = x²\n. Its slope at position\nx\nis\n2x\n. At\nx = 3\nthe slope is 6, steep and positive, so the loss climbs to the right and you should step left. At\nx = -2\nit is -4, so you step right. At\nx = 0\nit is 0, and you have arrived.\nimport\ntorch\nx\n=\ntorch\n.\ntensor\n(\n3.0\n,\nrequires_grad\n=\nTrue\n)\ny\n=\nx\n**\n2\ny\n.\nbackward\n()\nprint\n(\nx\n.\ngrad\n)\n# tensor(6.)\nEnter fullscreen mode\nExit fullscreen mode\nPyTorch produced the slope without being told the formula. That is autograd, and the rest of this article is how.\nTurning on the tape\nTensors do not track gradients by default. That would be wasted work on your input data, which is never adjusted. You opt in with\nrequires_grad=True\n:\nx\n=\ntorch\n.\ntensor\n(\n3.0\n,\nrequires_grad\n=\nTrue\n)\nprint\n(\nx\n.\nrequires_grad\n)\n# True\nEnter fullscreen mode\nExit fullscreen mode\nFrom this moment every operation involving\nx\ngets recorded. Think of it as a receipt. Each multiplication, addition and power you apply gets written down, along with enough information to reverse it later.\nThe computation graph\nThe receipt has a proper name, the computation graph. It is a chain of nodes where each node is an operation and each edge carries a tensor from one operation into the next.\nx\n=\ntorch\n.\ntensor\n(\n3.0\n,\nrequires_grad\n=\nTrue\n)\ny\n=\nx\n**\n2\n# node: power\nz\n=\n2\n*\ny\n+\n1\n# nodes: multiply, then add\nEnter fullscreen mode\nExit fullscreen mode\nThree lines of Python, four values, three operation nodes:\nx (leaf) **2 ×2 +1 z\n 3.0 → 9.0 → 18.0 → 19.0 → 19.0\nEnter fullscreen mode\nExit fullscreen mode\nThis is the forward pass. It runs left to right, computes the answer, and as a side effect builds the graph. Nothing has been differentiated yet. The graph exists purely so that something can walk it backwards.\nTwo things are worth noticing here. First,\nx\nis a\nleaf\n— you created it directly rather than computing it from something else. Second, every result node remembers the operation that produced it. You can see this:\nprint\n(\nz\n.\ngrad_fn\n)\n# \nprint\n(\ny\n.\ngrad_fn\n)\n# \nprint\n(\nx\n.\ngrad_fn\n)\n# None ← leaves have no history\nEnter fullscreen mode\nExit fullscreen mode\ngrad_fn\nis the recording.\nAddBackward0\nis not the addition, it is the\ninstruction for reversing\nthe addition.\nCalling .backward()\nz\n.\nbackward\n()\nprint\n(\nx\n.\ngrad\n)\n# tensor(12.)\nEnter fullscreen mode\nExit fullscreen mode\nCheck it by hand.\nz = 2x² + 1\n, so\ndz/dx = 4x\n, and at\nx = 3\nthat is 12. Correct.\nBut the interesting part is not that the answer is right, it is how it was produced, because PyTorch never formed the expression\n4x\nat all. It walked the graph right to left and at each node multiplied the incoming gradient by that node's own local slope:\nz = 19 +1 ×2 **2 x.grad\nstart slope 1 slope 2 slope 2x = 6\ngrad 1 → grad 1 → grad 2 → grad 12\nEnter fullscreen mode\nExit fullscreen mode\nReading right to left: the\n+1\nnode has slope 1, so the gradient passes through unchanged. The\n×2\nnode has slope 2, so the gradient doubles. The\nx²\nnode has slope\n2x\n, which at\nx = 3\nis 6, so the gradient multiplies by 6. Altogether\n1 × 2 × 6 = 12\n.\nThat is the whole of backpropagation. It is the chain rule, applied one node at a time, right to left, automatically. Nobody ever writes down\n4x\n.\nThis also explains why the forward pass has to happen first, and why it has to store its intermediate values. The\nx²\nnode's local slope is\n2x\n, which needs the\nvalue\nof\nx\nthat went in. Every node keeps whatever it needs to compute its own derivative later. This is where the memory goes during training, and it is why a batch that fits in memory for inference can still run you out of memory when training.\nNow do it on an actual network, by hand\nEverything above is one variable. Here is the smallest thing that is honestly a neural network:\none input, one weight, one bias, a ReLU, and a squared-error loss. Five operations. Enough to show every part of the chain rule, small enough to hold in your head.\nNumbers:\nx = 2.0\n,\nw = 3.0\n,\nb = -1.0\n, target\ny = 2.0\n.\nForward pass\n, left to right, writing down everything:\nh = w·x + b = 3·2 + (-1) = 5.0\na = ReLU(h) = max(0, 5) = 5.0\nL = (a - y)² = (5 - 2)² = 9.0\nEnter fullscreen mode\nExit fullscreen mode\nLoss is 9. Now the question that matters: how should\nw\nand\nb\nchange to make it smaller?\nBackward pass\n, one node at a time, right to left. At each node, local derivative times the\ngradient arriving from the right.\nLoss to\na\n.\nL = (a - y)²\n, so\n∂L/∂a = 2(a - y) = 2(5 - 2) = 6.0\na\nto\nh\n, through the ReLU.\nReLU's derivative is 1 if its input was positive and 0 if it was\nnegative. Our\nh\nwas 5, positive, so\n∂a/∂h = 1.0\nand therefore\n∂L/∂h = 6.0 × 1.0 = 6.0\nh\nto\nw\nand\nb\n.\nSince\nh = w·x + b\n:\n∂h/∂w = x = 2.0 → ∂L/∂w = 6.0 × 2.0 = 12.0\n∂h/∂b = 1.0 → ∂L/∂b = 6.0 × 1.0 = 6.0\nEnter fullscreen mode\nExit fullscreen mode\nSo by hand:\n∂L/∂w = 12\nand\n∂L/∂b = 6\n.\nNow ask PyTorch the same question:\nx\n=\ntorch\n.\ntensor\n(\n2.0\n)\nw\n=\ntorch\n.\ntensor\n(\n3.0\n,\nrequires_grad\n=\nTrue\n)\nb\n=\ntorch\n.\ntensor\n(\n-\n1.0\n,\nrequires_grad\n=\nTrue\n)\ny\n=\ntorch\n.\ntensor\n(\n2.0\n)\nh\n=\nw\n*\nx\n+\nb\na\n=\ntorch\n.\nrelu\n(\nh\n)\nloss\n=\n(\na\n-\ny\n)\n**\n2\nloss\n.\nbackward\n()\nprint\n(\nf\n\"\nloss =\n{\nloss\n.\nitem\n()\n}\n\"\n)\n# 9.0\nprint\n(\nf\n\"\ndL/dw =\n{\nw\n.\ngrad\n.\nitem\n()\n}\n\"\n)\n# 12.0\nprint\n(\nf\n\"\ndL/db =\n{\nb\n.\ngrad\n.\nitem\n()\n}\n\"\n)\n# 6.0\nEnter fullscreen mode\nExit fullscreen mode\nExact match, all three. Autograd traced the same five operations and applied the same chain rule. It just did it without asking you.\nIf you read nothing else here, run that block and compare it to the hand calculation above. The whole point of the exercise is the moment the numbers agree.\nWhat happens when the ReLU is closed\nChange one number. Keep\nw = 3\nand\nb = -1\n, but feed\nx = -1.0\n. Now\nh = 3·(-1) + (-1) = -4\n, which is negative, so ReLU outputs 0 and its local derivative is also 0.\nx\n=\ntorch\n.\ntensor\n(\n-\n1.0\n)\nw\n=\ntorch\n.\ntensor\n(\n3.0\n,\nrequires_grad\n=\nTrue\n)\nb\n=\ntorch\n.\ntensor\n(\n-\n1.0\n,\nrequires_grad\n=\nTrue\n)\ny\n=\ntorch\n.\ntensor\n(\n2.0\n)\nh\n=\nw\n*\nx\n+\nb\n# -4.0\na\n=\ntorch\n.\nrelu\n(\nh\n)\n# 0.0\nloss\n=\n(\na\n-\ny\n)\n**\n2\n# (0 - 2)² = 4.0\nloss\n.\nbackward\n()\nprint\n(\nw\n.\ngrad\n.\nitem\n())\n# 0.0\nprint\n(\nb\n.\ngrad\n.\nitem\n())\n# 0.0\nEnter fullscreen mode\nExit fullscreen mode\nThe loss is 4, so the network is wrong, and yet both gradients are exactly zero. No learning\nsignal reaches\nw\nor\nb\nat all.\nThis follows straight from the hand calculation.\n∂a/∂h = 0\n, and every gradient behind that node gets multiplied by zero on its way through. The ReLU is a gate, and this one is shut.\nThat is the dying-ReLU problem, and it is worth meeting it here rather than three months later in a model with fifty layers. When a neuron's input is negative for every example in your data, its gradient is permanently zero and it never learns again.\nLeakyReLU\nexists because it lets a small gradient through instead.\nMore generally: when a model stops learning and you cannot see why, the question to ask is what is multiplying the gradient by zero on the way back.\nThree mechanics that catch nearly everyone\n1. Only leaves keep their\n.grad\nIntermediate results do not store their gradient. It is computed, used to keep the chain going, and thrown away, because keeping every intermediate gradient in a real model would be enormous. If you want one, ask before calling backward:\nx\n=\ntorch\n.\ntensor\n(\n2.0\n,\nrequires_grad\n=\nTrue\n)\ny\n=\nx\n**\n3\ny\n.\nretain_grad\n()\n# keep y's gradient too\nz\n=\ny\n*\n5\nz\n.\nbackward\n()\nprint\n(\nx\n.\ngrad\n)\n# tensor(60.) dz/dx = 15x² = 60\nprint\n(\ny\n.\ngrad\n)\n# tensor(5.) dz/dy = 5\nEnter fullscreen mode\nExit fullscreen mode\nWithout\nretain_grad()\n,\ny.grad\nis\nNone\nand you get a warning rather than an error, which is why people spend twenty minutes confused by it.\n2. Gradients accumulate, they do not overwrite\nThis one causes more silently broken training loops than anything else in PyTorch.\nx\n=\ntorch\n.\ntensor\n(\n3.0\n,\nrequires_grad\n=\nTrue\n)\nfor\ni\nin\nrange\n(\n3\n):\ny\n=\nx\n**\n2\ny\n.\nbackward\n()\nprint\n(\nf\n\"\nstep\n{\ni\n}\ngrad =\n{\nx\n.\ngrad\n.\nitem\n()\n}\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nstep 0 grad = 6.0\nstep 1 grad = 12.0\nstep 2 grad = 18.0\nEnter fullscreen mode\nExit fullscreen mode\nThe gradient of\nx²\nat\nx = 3\nis 6, every time. It reads 12 and then 18 because\n.grad\nis added to, not replaced.\nThis is deliberate. It is what lets you accumulate gradients over several mini-batches and take one larger step, which is how people train with an effective batch size their GPU cannot hold.\nBut it means that unless you clear it, every step of your training loop is stepping on a sum of all previous gradients, and your loss curve will do something strange that is very hard to diagnose from the outside.\nThe fix is\nx.grad.zero_()\n, or in a real loop,\noptimizer.zero_grad()\nbefore every\nloss.backward()\n. That line is not boilerplate. It is load-bearing.\n3.\nno_grad\nand\ndetach\n— switching the recorder off\nDuring evaluation you do not need gradients, and building the graph costs both time and memory.\nx\n=\ntorch\n.\ntensor\n(\n3.0\n,\nrequires_grad\n=\nTrue\n)\nwith\ntorch\n.\nno_grad\n():\ny\n=\nx\n**\n2\nprint\n(\ny\n.\nrequires_grad\n)\n# False\nEnter fullscreen mode\nExit fullscreen mode\nInside the block no graph is built. This is why every evaluation loop you have ever copied is wrapped in\ntorch.no_grad()\n.\n.detach()\nis the narrower tool. It gives you a tensor that shares the same data but has no connection to the graph:\nx\n=\ntorch\n.\ntensor\n(\n3.0\n,\nrequires_grad\n=\nTrue\n)\ny\n=\nx\n**\n2\ny_val\n=\ny\n.\ndetach\n()\nprint\n(\ny_val\n)\n# tensor(9.)\nprint\n(\ny_val\n.\nrequires_grad\n)\n# False\nEnter fullscreen mode\nExit fullscreen mode\nUse it when you want a value without dragging its history along — logging, or using a model's output as a target that should not be differentiated through. It is the mechanism behind stop-gradient tricks and target networks, and it is also, occasionally, the reason your gradient is unexpectedly zero.\nOne call, every parameter\nEverything so far had one or two parameters so the arithmetic stayed visible. Nothing changes at scale.\nw\n=\ntorch\n.\ntensor\n(\n2.0\n,\nrequires_grad\n=\nTrue\n)\nb\n=\ntorch\n.\ntensor\n(\n1.0\n,\nrequires_grad\n=\nTrue\n)\nx\n=\ntorch\n.\ntensor\n(\n3.0\n)\ny\n=\nw\n*\nx\n+\nb\n# 2·3 + 1 = 7\nloss\n=\n(\ny\n-\n5\n)\n**\n2\n# (7 - 5)² = 4\nloss\n.\nbackward\n()\nprint\n(\nw\n.\ngrad\n)\n# tensor(12.)\nprint\n(\nb\n.\ngrad\n)\n# tensor(4.)\nEnter fullscreen mode\nExit fullscreen mode\nOne\nbackward()\nfilled in both. In a model with eleven million parameters, the same single call fills in all eleven million, because the graph reaches every one of them and the chain rule multiplies along every path. Nothing about the mechanism is different. There are just more nodes.\nThe summary worth keeping\nrequires_grad=True\nmarks a tensor as something you want the gradient of. Model parameters\nget it automatically; your data does not need it.\nThe forward pass computes the answer and records a graph as a side effect. Each node stores what it needs to reverse itself later.\n.backward()\nwalks that graph in reverse, multiplying the incoming gradient by each node's\nlocal derivative. That is the chain rule, and it is all backpropagation is.\nGradients land in\n.grad\non leaf tensors only. Use\nretain_grad()\nfor intermediates.\nGradients accumulate.\nzero_grad()\nbefore every backward, always.\ntorch.no_grad()\nfor evaluation,\n.detach()\nfor a single value without its history.\nA zero gradient means something on the path back multiplied by zero. A closed ReLU is the\nusual suspect.\nTry this before you close the tab\nTake\nx = 4.0\nwith\nrequires_grad=True\n, compute\ny = x**3 - 2*x\n, call\nbackward()\n, and print\nx.grad\n. Then work out\n3x² - 2\nat\nx = 4\non paper and check whether they agree. It takes\nabout ninety seconds and it is worth more than rereading this article.\nThen the harder version: put a second weight after the ReLU in the tiny network above,\nw2 =\n0.5\n, and write out the full chain rule for\n∂L/∂w\nbefore you run it.\nThis is one chapter's worth of an idea from my book,\nPyTorch From Ground Up\n,\nwhich builds\neverything from tensors upward so nothing stays vague. If it helped:\n8 chapters are free, no email required\n,\nthere's a\nfree one-page tensor cheat-sheet here\n, every example runs\nin the\ncompanion notebooks on GitHub\n, and\nthe full book is on\nLeanpub\nor in\npaperback and Kindle on Amazon\n.\nMore in this series\nShape mechanics, the part that has to be solid before any of this makes sense:\nReshape vs View in PyTorch\nPyTorch Broadcasting Explained", "metadata": { "title": "PyTorch Autograd Explained: What .backward() Actually Does - DEV Community", "description": "most autograd tutorials show you requires_grad, then .backward(), then .grad, and then they hand you... Tagged with deeplearning, machinelearning, python, tutorial.", "keywords": "deeplearning, machinelearning, python, tutorial, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/pytorchfromgroundup/pytorch-autograd-explained-what-backward-actually-does-4acg", "og:title": "PyTorch Autograd Explained: What .backward() Actually Does", "og:description": "most autograd tutorials show you requires_grad, then .backward(), then .grad, and then they hand you...", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4v1rjem6rk14uyby1irs.png", "twitter:site": "@thepracticaldev", "twitter:creator": "@wesamkhallaf", "twitter:title": "PyTorch Autograd Explained: What .backward() Actually Does", "twitter:description": "most autograd tutorials show you requires_grad, then .backward(), then .grad, and then they hand you...", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4v1rjem6rk14uyby1irs.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/869bb105719b/code/body-000.txt", "chars": 112, "language": "text" }, { "file": "attachments/869bb105719b/code/body-001.txt", "chars": 75, "language": "text" }, { "file": "attachments/869bb105719b/code/body-002.txt", "chars": 120, "language": "text" }, { "file": "attachments/869bb105719b/code/body-003.txt", "chars": 102, "language": "text" }, { "file": "attachments/869bb105719b/code/body-004.txt", "chars": 158, "language": "text" }, { "file": "attachments/869bb105719b/code/body-005.txt", "chars": 45, "language": "text" }, { "file": "attachments/869bb105719b/code/body-006.txt", "chars": 187, "language": "text" }, { "file": "attachments/869bb105719b/code/body-007.txt", "chars": 108, "language": "text" }, { "file": "attachments/869bb105719b/code/body-008.txt", "chars": 97, "language": "text" }, { "file": "attachments/869bb105719b/code/body-009.txt", "chars": 327, "language": "text" }, { "file": "attachments/869bb105719b/code/body-010.txt", "chars": 306, "language": "text" }, { "file": "attachments/869bb105719b/code/body-011.txt", "chars": 218, "language": "text" }, { "file": "attachments/869bb105719b/code/body-012.txt", "chars": 141, "language": "text" }, { "file": "attachments/869bb105719b/code/body-013.txt", "chars": 59, "language": "text" }, { "file": "attachments/869bb105719b/code/body-014.txt", "chars": 117, "language": "text" }, { "file": "attachments/869bb105719b/code/body-015.txt", "chars": 154, "language": "text" }, { "file": "attachments/869bb105719b/code/body-016.txt", "chars": 263, "language": "text" } ], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://github.com/pytorch-from-ground-up/book_code/blob/main/PyTorch-From-Ground-Up-8-Free-Chapters.pdf", "label": "8 chapters are free, no email required", "type": "a" }, { "href": "https://payhip.com/b/7ukxh", "label": "free one-page tensor cheat-sheet here", "type": "a" }, { "href": "https://github.com/pytorch-from-ground-up/book_code", "label": "companion notebooks on GitHub", "type": "a" }, { "href": "https://leanpub.com/pytorchfromgroundup", "label": "Leanpub", "type": "a" }, { "href": "https://www.amazon.com/dp/B0H8WMCV33", "label": "paperback and Kindle on Amazon", "type": "a" }, { "href": "https://dev.to/pytorchfromgroundup/reshape-vs-view-in-pytorch-whats-the-difference-and-when-view-breaks-4a9o", "label": "Reshape vs View in PyTorch", "type": "a" }, { "href": "https://dev.to/pytorchfromgroundup/pytorch-broadcasting-explained-the-3-rules-and-the-silent-bug-that-bites-everyone-3606", "label": "PyTorch Broadcasting Explained", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:48:51Z", "source_url": "https://dev.to/pytorchfromgroundup/pytorch-autograd-explained-what-backward-actually-does-4acg", "papers": [ { "status": "ok", "url": "https://raw.githubusercontent.com/pytorch-from-ground-up/book_code/main/PyTorch-From-Ground-Up-8-Free-Chapters.pdf", "bytes": 2552607, "reused": true, "local_path": "papers/repair/2927ef550363909d894f.pdf" } ], "actual_video_files": [], "body_chars": 13722, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/869bb105719b/pages/source.html", "local_html": "attachments/869bb105719b/pages/article.html", "body_base_path": "attachments/869bb105719b/pages/", "body_markdown_path": "attachments/869bb105719b/pages/article.md" }