{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Wesam Khallaf",
  "title_listed": "What optimizer.step() Actually Does: One Step of SGD, Momentum and Adam by Hand",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/pytorchfromgroundup/what-optimizerstep-actually-does-one-step-of-sgd-momentum-and-adam-by-hand-44fe",
  "publish_date": "2026-08-28",
  "corpus_stage": "",
  "notes": "",
  "id": "8438dbe1f5c1",
  "collected_at": "2026-10-02T13:53:54",
  "title": "What optimizer.step() Actually Does: One Step of SGD, Momentum and Adam by Hand - DEV Community",
  "body_markdown": "last time i took a two-layer network and computed all four gradients by hand, and then ran the same thing in pytorch and autograd landed on the same four numbers,\n30, 30, 20, 10\n. and then i stopped there, which is exactly one line too early, because in a real training loop the next thing that happens is\noptimizer.step()\nand that line is where the numbers actually turn into learning. so this time i want to do the same trick with the optimizer, take those four gradients and apply one update by hand, first with plain SGD, then with momentum, then with Adam, and see what each of them does to the same starting point. the whole thing fits in a few lines of arithmetic and it removes away a lot of vague feeling about learning rates, because you get to watch the same network survive one step and die on another.\nWhere we are\nsame network as last time, two layers, one hidden ReLU, squared error:\nx ──▶ h1 = w1·x + b1 ──▶ a1 = ReLU(h1) ──▶ h2 = w2·a1 + b2 ──▶ L = (h2 − y)²\nEnter fullscreen mode\nExit fullscreen mode\nx  = 1.0        y  = 2.0\nw1 = 2.0        b1 = 0.0\nw2 = 3.0        b2 = 1.0\nEnter fullscreen mode\nExit fullscreen mode\nforward gives\nh1 = 2\n,\na1 = 2\n,\nh2 = 7\n,\nL = 25\n, and backward gives\n∂L/∂w1 = 30    ∂L/∂b1 = 30    ∂L/∂w2 = 20    ∂L/∂b2 = 10\nEnter fullscreen mode\nExit fullscreen mode\nan optimizer's entire job is to take that list of four numbers and decide how far, and in what direction, each of the four parameters moves. that is all it is. it does not see the network, it does not know there is a ReLU in there, it gets a parameter and the gradient sitting on it and nothing else.\nSGD, which is one line of arithmetic\nthe rule is\nparam = param − lr × grad\n, and there is nothing hidden in it. with\nlr = 0.1\n:\nw1 = 2.0 − 0.1·30 = −1.0\nb1 = 0.0 − 0.1·30 = −3.0\nw2 = 3.0 − 0.1·20 =  1.0\nb2 = 1.0 − 0.1·10 =  0.0\nEnter fullscreen mode\nExit fullscreen mode\nimport\ntorch\ndef\nnet\n(\nw1\n,\nb1\n,\nw2\n,\nb2\n,\nx\n=\ntorch\n.\ntensor\n(\n1.0\n),\ny\n=\ntorch\n.\ntensor\n(\n2.0\n)):\nh1\n=\nw1\n*\nx\n+\nb1\na1\n=\ntorch\n.\nrelu\n(\nh1\n)\nh2\n=\nw2\n*\na1\n+\nb2\nreturn\n(\nh2\n-\ny\n)\n**\n2\n,\nh1\np\n=\n[\ntorch\n.\ntensor\n(\nv\n,\nrequires_grad\n=\nTrue\n)\nfor\nv\nin\n(\n2.0\n,\n0.0\n,\n3.0\n,\n1.0\n)]\nopt\n=\ntorch\n.\noptim\n.\nSGD\n(\np\n,\nlr\n=\n0.1\n)\nloss\n,\nh1\n=\nnet\n(\n*\np\n)\nopt\n.\nzero_grad\n()\nloss\n.\nbackward\n()\nopt\n.\nstep\n()\nprint\n([\nround\n(\nt\n.\nitem\n(),\n4\n)\nfor\nt\nin\np\n])\n# [-1.0, -3.0, 1.0, 0.0]\nprint\n(\nnet\n(\n*\np\n)[\n0\n].\nitem\n(),\nnet\n(\n*\np\n)[\n1\n].\nitem\n())\n# 4.0   -4.0\nEnter fullscreen mode\nExit fullscreen mode\nnow look at what that did. the loss went from 25 to 4, which reads like a great first step. but the new\nh1\nis\n−4\n, and a ReLU with a negative input outputs zero and has derivative zero, so the hidden\nneuron is dead, and from the previous article we know exactly what that means for the gradients,\nw1\n,\nb1\nand\nw2\nall get zero from here on and only\nb2\nkeeps learning. one step. the network\nlost its hidden layer on the very first update and the loss number went down while it happened, so the loss will not tell you.\nsame thing at\nlr = 0.01\n:\nw1 = 1.7    b1 = −0.3    w2 = 2.8    b2 = 0.9\nh1 = 1.4    loss = 7.9524\nEnter fullscreen mode\nExit fullscreen mode\nstill alive, loss 25 → 7.95, and it keeps going down from there and reaches basically zero after about thirty steps with\nh1\nstill positive at 0.48. so the difference between a network that trains\nand a network that quietly stopped having a hidden layer, on this problem, is one number in the constructor. that is why the learning rate is the hyperparameter people actually tune, and why \"the\nloss went down\" is not evidence that anything is healthy.\nMomentum, and the part where it does not do what you expect\npytorch writes momentum like this, and it is worth knowing the exact form because a lot of textbooks put the learning rate inside the buffer but pytorch does not:\nbuf = momentum · buf + grad        (on the first step, buf = grad)\nparam = param − lr · buf\nEnter fullscreen mode\nExit fullscreen mode\nso the buffer is a running sum of gradients with the old ones decayed by 0.9 each step, and the update uses that instead of the raw gradient. run it on our network with\nlr = 0.01, momentum = 0.9\nand do the first two steps by hand.\nstep 1.\nthe buffer starts empty, so\nbuf = grad = [30, 30, 20, 10]\nand the update is identical to plain SGD. momentum does literally nothing on the first step, it has no history yet.\nw1 = 1.7    b1 = −0.3    w2 = 2.8    b2 = 0.9      loss = 7.9524\nEnter fullscreen mode\nExit fullscreen mode\nstep 2.\ngradients at the new point are\n[15.792, 15.792, 7.896, 5.64]\n. the buffer is\nbuf = 0.9·[30, 30, 20, 10] + [15.792, 15.792, 7.896, 5.64]\n    = [42.792, 42.792, 25.896, 14.64]\nEnter fullscreen mode\nExit fullscreen mode\n42.792\nwhere the gradient alone is\n15.792\n, so the step is nearly three times bigger than the\ngradient asked for, because the previous gradient pointed the same way and momentum is adding them\nup. and it pays off immediately:\nw1 = 1.27208   b1 = −0.72792   w2 = 2.54104   b2 = 0.7536\nh1 = 0.54416   loss = 0.018587\nEnter fullscreen mode\nExit fullscreen mode\nplain SGD at the same step 2 is at\n3.217154\n. momentum is at\n0.0186\n. that is the case for it in one line, on a slope where the gradients keep pointing the same direction it accumulates and covers\nground much faster than the raw gradient would.\nstep 3, where it goes wrong.\nwe overshot.\nh1\nwas\n0.544\nand the buffer is still carrying\n39.2\nof leftover velocity, so the next update pushes\nh1\nto\n−0.239953\nand the ReLU shuts. loss back up to\n1.906848\n. and here is the thing that makes momentum worth understanding rather than just switching on, at step 4 the gradients on\nw1\nand\nb1\nare exactly\n0\n, the neuron is dead, and the\nbuffer is still\n35.2851\n, so\nthe parameters keep moving anyway\n. that is what momentum means, taken literally. no gradient, still moving.\nh1\ngoes to\n−0.9457\n, then\n−1.58\n, and it is being\ndriven further into the dead zone by the memory of gradients from three steps ago.\nrun it out to 60 steps and the ending is the bit i would not have guessed:\nmomentum, 60 steps:  loss = 0.005930    w1 = −2.6398  b1 = −4.6398  w2 = 0.2008  b2 = 1.9230\nplain SGD, 30 steps: loss ≈ 0.000000    w1 =  1.2415  b1 = −0.7585  w2 = 2.6270  b2 = 0.7313\nEnter fullscreen mode\nExit fullscreen mode\nthe momentum run reports a loss of\n0.0059\nand looks converged. but\nh1 = −2.6398 − 4.6398 = −7.28\n,\nthe hidden neuron has been dead for fifty-odd steps,\nw2\ndecayed to\n0.2\nbecause it is reading from a constant zero, and the whole thing is now the output bias\nb2 = 1.923\npredicting a target of\n2\n.\nit solved the problem by throwing the network away and keeping the bias. a low loss on a tiny problem can mean the model found the answer or it can mean the model routed around its own dead parts, and\nthe only way to see which one you have is to print something other than the loss.\ni am not saying do not use momentum, on real problems with mini-batch noise it is almost always better and it costs one keyword. i am saying the failure mode is specific and worth having seen once,\nbecause \"momentum keeps moving when the gradient is zero\" is the same property that gets you through a flat region and the same property that drives a neuron off a cliff.\nAdam, and what its learning rate actually is\nAdam keeps two running averages per parameter, one of the gradient and one of the squared gradient, then divides one by the square root of the other:\nm = β1·m + (1−β1)·g              β1 = 0.9      (average gradient)\nv = β2·v + (1−β2)·g²             β2 = 0.999    (average squared gradient)\n\nm̂ = m / (1 − β1^t)               bias correction, t = step number\nv̂ = v / (1 − β2^t)\n\nparam = param − lr · m̂ / (√v̂ + ε)\nEnter fullscreen mode\nExit fullscreen mode\nthe bias correction exists because\nm\nand\nv\nstart at zero, so without it the first few steps would be far too small. now do step 1 by hand and watch something nice happen.\nm\nand\nv\nare both zero, so\nm = 0.1·g      m̂ = m / (1 − 0.9)   = g\nv = 0.001·g²   v̂ = v / (1 − 0.999) = g²\nEnter fullscreen mode\nExit fullscreen mode\nso\nm̂ / √v̂ = g / |g| = 1\n, for every parameter, whatever its gradient was. the update is just\nlr\n.\nfor our four gradients\n30, 30, 20, 10\nat\nlr = 1e-3\n:\nw1 = 1.999    b1 = −0.001    w2 = 2.999    b2 = 0.999      loss = 24.910101\nEnter fullscreen mode\nExit fullscreen mode\nall four moved by exactly\n0.001\n. the one with a gradient of 30 and the one with a gradient of 10 moved the same distance. i checked this to twelve decimals out of suspicion and it is\n0.001000000000\nfor three of them and\n0.000999999999\nfor the fourth, the difference being the\nε = 1e-8\nin the denominator.\nthat is the single most useful thing to know about Adam.\nits learning rate is a step size, not a multiplier on the gradient.\nin SGD,\nlr = 1e-3\nmeans \"move a thousandth of whatever the gradient\nsays\", so the actual distance depends on gradient scale, which depends on your loss and your data and your initialization. in Adam,\nlr = 1e-3\nmeans \"move about\n0.001\n\", full stop. that is why\n1e-3\nis a sane default across wildly different models and why the same value in SGD would be useless on our problem here.\nit also means Adam is not automatically faster:\nAdam lr=1e-3,  60 steps: loss = 20.04    (still crawling, it moves 0.001 per step)\nAdam lr=0.1,   60 steps: loss = 0.0146\nplain SGD lr=0.01, 30 steps: loss ≈ 0\nEnter fullscreen mode\nExit fullscreen mode\non four parameters whose gradients are all roughly the same size, adapting per parameter buys you nothing and the fixed step size just makes it slow. so where does it earn its place.\nThe experiment where Adam is obviously right\nchange one thing, set\nx = 0.01\ninstead of\n1.0\n. nothing else. now the gradients are\n∂L/∂w1 = −0.0564    ∂L/∂b1 = −5.64    ∂L/∂w2 = −0.0376    ∂L/∂b2 = −1.88\nEnter fullscreen mode\nExit fullscreen mode\nw1\nand\nb1\nsit in the same layer and their gradients are a hundred times apart, because\n∂L/∂w1 = ∂L/∂h1 · x\nand\n∂L/∂b1 = ∂L/∂h1 · 1\n, and\nx\nis\n0.01\n. this is not a contrived setup, it is what a badly scaled input feature does to you, and in a real network the same spread happens between embedding weights, biases and normalization scales.\ntrain it 200 steps both ways and look at how far each parameter actually travelled:\nSGD  lr=0.01 :  w1 moved 0.0028      b1 moved 0.2808      (ratio 100)\nAdam lr=0.01 :  w1 moved 0.2207      b1 moved 0.2207      (ratio 1)\nEnter fullscreen mode\nExit fullscreen mode\nSGD's single learning rate gets divided among the parameters by their gradient magnitudes, so\nw1\nbarely moves at all and effectively is not being trained, while\nb1\ndoes all the work. Adam moved\nboth by the identical amount, because dividing by\n√v̂\ncancels the scale of each parameter's own gradient. that is what \"per-parameter learning rates\" means concretely, and it is the reason it is\nthe default for anything with heterogeneous parameter groups in it.\nboth runs reach a near-zero loss here, this network is too small for it to matter. the point is not which one won, the point is what the movement numbers say about where the learning rate went.\nThe two lines, and the order\nwhichever one you pick, the loop is the same three calls and swapping optimizer changes nothing else:\nfor\nepoch\nin\nrange\n(\n200\n):\npred\n=\nmodel\n(\nx\n)\nloss\n=\nloss_fn\n(\npred\n,\ny\n)\noptimizer\n.\nzero_grad\n()\n# clear last step's gradients\nloss\n.\nbackward\n()\n# compute this step's gradients\noptimizer\n.\nstep\n()\n# apply the update rule\nEnter fullscreen mode\nExit fullscreen mode\nthe order is load-bearing.\nstep()\nbefore\nbackward()\nupdates the parameters with last step's gradients, and the loop still runs and the loss still prints, it is just always one step stale.\nskipping\nzero_grad()\nis worse, gradients accumulate rather than overwrite, which i wrote about in the autograd piece, and the effective step grows every iteration until the thing diverges. neither of those raises an error. they just train badly, which is the pattern with most of this.\nTry it before you close the tab\ntake the network at the top and run one SGD step at\nlr = 0.05\ninstead of\n0.1\nor\n0.01\n, and work out\nh1\nbefore you run it. predict whether the hidden neuron survives. there is a specific learning rate where it dies and you can solve for it in one line,\nh1 = (2 − 30·lr) + (0 − 30·lr)\n, so it goes\nnegative at\nlr = 1/30\n. anything above\n0.0333\nkills the neuron on the first step. finding that number yourself is worth more than the rest of this article.\nthen set\nmomentum = 0.9\nand print\nh1\nat every step for ten steps, and watch it cross zero and keep going while the gradient reads\n0.0\n.\nwhat actually made you change optimizer the last time you changed it?\nfor me it has mostly been switching to Adam because something would not train and then never switching back, which is not\nreally a reason, and the honest answer is i did not look at the gradient scales until i started writing this chapter.\nThis is one chapter's worth of an idea from my book,\nPyTorch From Ground Up\n,\nwhich builds everything from tensors upward so nothing stays vague. If it helped:\n8 chapters are free, no email required\n,\nthere's a\nfree one-page tensor cheat-sheet here\n, every example runs in\nthe\ncompanion notebooks on GitHub\n, and the\nfull book is on\nLeanpub\nor in\npaperback and Kindle on Amazon\n.\nMore in this series\nHow Training Actually Works\n, the part where the training loop stops being magic:\nPyTorch Autograd Explained: What\n.backward()\nActually Does\nBackpropagation by Hand: Two Layers, a Pen, and Then Autograd Agrees\nThe shape mechanics underneath all of it, worth having solid first:\nReshape vs View in PyTorch\nPyTorch Broadcasting Explained\nComing next in\nHow Training Actually Works\n: what the loss function is actually choosing for you.",
  "metadata": {
    "title": "What optimizer.step() Actually Does: One Step of SGD, Momentum and Adam by Hand - DEV Community",
    "description": "last time i took a two-layer network and computed all four gradients by hand, and then ran the same... Tagged with pytorch, deeplearning, machinelearning, beginners.",
    "keywords": "pytorch, deeplearning, machinelearning, beginners, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/pytorchfromgroundup/what-optimizerstep-actually-does-one-step-of-sgd-momentum-and-adam-by-hand-44fe",
    "og:title": "What optimizer.step() Actually Does: One Step of SGD, Momentum and Adam by Hand",
    "og:description": "last time i took a two-layer network and computed all four gradients by hand, and then ran the same...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3tvbb7xoj35i4butv8pr.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@wesamkhallaf",
    "twitter:title": "What optimizer.step() Actually Does: One Step of SGD, Momentum and Adam by Hand",
    "twitter:description": "last time i took a two-layer network and computed all four gradients by hand, and then ran the same...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3tvbb7xoj35i4butv8pr.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/8438dbe1f5c1/code/body-000.txt",
      "chars": 77,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-001.txt",
      "chars": 75,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-002.txt",
      "chars": 57,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-003.txt",
      "chars": 100,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-004.txt",
      "chars": 471,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-005.txt",
      "chars": 72,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-006.txt",
      "chars": 92,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-007.txt",
      "chars": 65,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-008.txt",
      "chars": 97,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-009.txt",
      "chars": 89,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-010.txt",
      "chars": 186,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-011.txt",
      "chars": 265,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-012.txt",
      "chars": 81,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-013.txt",
      "chars": 76,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-014.txt",
      "chars": 160,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-015.txt",
      "chars": 73,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-016.txt",
      "chars": 138,
      "language": "text"
    },
    {
      "file": "attachments/8438dbe1f5c1/code/body-017.txt",
      "chars": 244,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/pytorch-from-ground-up/book_code/blob/main/PyTorch-From-Ground-Up-8-Free-Chapters.pdf",
      "label": "8 chapters are free, no email required",
      "type": "a"
    },
    {
      "href": "https://payhip.com/b/7ukxh",
      "label": "free one-page tensor cheat-sheet here",
      "type": "a"
    },
    {
      "href": "https://github.com/pytorch-from-ground-up/book_code",
      "label": "companion notebooks on GitHub",
      "type": "a"
    },
    {
      "href": "https://leanpub.com/pytorchfromgroundup",
      "label": "Leanpub",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/dp/B0H8WMCV33",
      "label": "paperback and Kindle on Amazon",
      "type": "a"
    },
    {
      "href": "https://dev.to/pytorchfromgroundup/pytorch-autograd-explained-what-backward-actually-does-4acg",
      "label": "PyTorch Autograd Explained: What .backward() Actually Does",
      "type": "a"
    },
    {
      "href": "https://dev.to/pytorchfromgroundup/backpropagation-by-hand-two-layers-a-pen-and-then-autograd-agrees-13i6",
      "label": "Backpropagation by Hand: Two Layers, a Pen, and Then Autograd Agrees",
      "type": "a"
    },
    {
      "href": "https://dev.to/pytorchfromgroundup/reshape-vs-view-in-pytorch-whats-the-difference-and-when-view-breaks-4a9o",
      "label": "Reshape vs View in PyTorch",
      "type": "a"
    },
    {
      "href": "https://dev.to/pytorchfromgroundup/pytorch-broadcasting-explained-the-3-rules-and-the-silent-bug-that-bites-everyone-3606",
      "label": "PyTorch Broadcasting Explained",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:51Z",
  "source_url": "https://dev.to/pytorchfromgroundup/what-optimizerstep-actually-does-one-step-of-sgd-momentum-and-adam-by-hand-44fe",
  "papers": [
    {
      "status": "ok",
      "url": "https://raw.githubusercontent.com/pytorch-from-ground-up/book_code/main/PyTorch-From-Ground-Up-8-Free-Chapters.pdf",
      "bytes": 2552607,
      "reused": true,
      "local_path": "papers/repair/2927ef550363909d894f.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 13654,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/8438dbe1f5c1/pages/source.html",
  "local_html": "attachments/8438dbe1f5c1/pages/article.html",
  "body_base_path": "attachments/8438dbe1f5c1/pages/",
  "body_markdown_path": "attachments/8438dbe1f5c1/pages/article.md"
}