{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Seth Wheeler",
  "title_listed": "Error Feedback, Gradient Compression, and Why Adam Breaks It",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/megapixel99/error-feedback-gradient-compression-and-why-adam-breaks-it-pm4",
  "publish_date": "2026-08-21",
  "corpus_stage": "",
  "notes": "",
  "id": "ae4879c46eba",
  "collected_at": "2026-10-02T13:52:04",
  "title": "Error Feedback, Gradient Compression, and Why Adam Breaks It - DEV Community",
  "body_markdown": "Error feedback is the oldest trick in lossy numerics wearing a machine-learning hat. You compress something, you keep the part you threw away, and you add the leftover back into the next thing you compress. Kahan summation does it for floating-point addition; delta-sigma converters do it for audio. In gradient compression it does something specific and valuable: each individual step stays biased, but the bias cancels across steps, so the sum of what the optimizer actually applied converges to the sum of the true gradients.\nThat guarantee is why error feedback shows up in every low-precision training recipe, including the one I wrote. It is also why I never questioned it. The guarantee is real, it is provable, and it holds under SGD.\nIt holds under SGD because SGD is\nlinear\nin the gradient. Adam is not, and it turns out that is not a footnote. Composed naively with Adam, error feedback in my measurements is not merely useless, it is worse than skipping the correction entirely:\n1.9 times further from the optimum than plain quantization, and 2.1 times further than not quantizing at all.\nThen the fix I published for it, when I finally ran the ablation that isolates it, turned out to be measuring something other than what I said it was.\nThe code for this study isn't public, so this post carries the numbers instead. A companion write-up (\"My gradient quantizer won on 45 of 45 tensors\") covers the scale rule and the noise floor this project mostly existed to measure. The correction below revises one paragraph of it, and I'll come to that.\nThe harness, and why it's deliberately crude\nAt four bits, everything in this project landed inside seed noise, which is a separate story with its own post. So this audit runs where the effect is large enough to see: a\n3-level\nquantizer (values snap to\n{-1, 0, +1}\ntimes one scale per block of 16) with a\nhard clip at 0.35 of the block maximum\n. That setting throws away a lot. Measured on Gaussian blocks, it has a relative MSE of\n28.2%\nand clips\n29.7%\nof all values, against roughly\n0.7%\nrelative MSE for NVFP4. It is about 40 times cruder than the format I actually care about, and that is the point: error feedback exists to clean up exactly this kind of systematic clipping bias, so a harsh clip is the setting where feedback should shine hardest.\nThe problem is convex least squares, 64 inputs by 32 outputs, 4,096 samples, minibatches of 128, constant learning rate, 4,000 steps. The metric is squared distance to the exact least-squares optimum (available in closed form, so there is no ambiguity about where the run should end up), averaged over the last 1,000 steps to read steady state rather than a transient, over 4 seeds. Every ordering I claim below holds at three sigma of the across-seed spread.\nConvex, tiny, and coarse. Hold that thought for the caveats section, because it constrains what the result licenses.\nUnder SGD it is textbook\narm\ndistance to optimum (SGD)\nfull precision\n2.33e-5\nquantized\n1.56e-5\nquantized + error feedback\n2.31e-5\nFeedback puts the quantized run back on the full-precision trajectory to three digits: 2.31e-5 against 2.33e-5. This is what the theory promises and it is satisfying to watch.\nThe quantized arm apparently\nbeating\nfull precision is a confound rather than a win. Hard clipping shrinks the effective step size, and on a constant-learning-rate convex problem a smaller step means a tighter steady-state ball around the optimum. It is the same reason a learning-rate sweep on this kind of objective always \"prefers\" the smallest rate offered. That is why every comparison here stays inside one optimizer, never across.\nUnder Adam it is a landmine\nSame quantizer, same clip, same seeds, same metric. Only the update rule changes.\narm\ndistance to optimum (Adam)\ntail loss\nfull precision\n8.86e-5\n0.000483\nquantized\n9.58e-5\n0.000490\nquantized + error feedback\n1.86e-4\n0.000582\nQuantization alone costs about 8% in distance, which is unremarkable. Adding the correction costs\n94% more than not adding it\n, and lands 2.1 times further out than the full-precision run. On steady-state loss it is +20.5% against full precision, where plain quantization was +1.4%. Read as excess distance over full precision, which is the quantity a correction exists to shrink, quantization's cost goes from 7.2e-6 to 9.74e-5: the one intervention whose entire purpose is to remove quantization's bias made quantization\n13.5 times more damaging\n.\nThe mechanism has two halves, and both come from the same source: Adam's update is\nm / sqrt(v)\n, so the gradient's magnitude is divided out.\nThe telescoping argument dies.\nError feedback's guarantee is a statement about a sum: residual added at step\nt\ncancels residual removed at step\nt-1\n, so what you applied over the whole run equals what you should have applied. That cancellation requires the map from gradient to parameter change to be linear. Adam's is not, so the residual is not re-applied with the same weight it was withheld. It is re-applied divided by whatever\nsqrt(v)\nhappens to be several steps later, which no longer has any reason to match.\nAnd the residual poisons the preconditioner.\nThe quantity fed into Adam is\ngradient + residual\n, and\nv\nis an average of its\nsquare\n. The carried residual is not signal; it is accumulated rounding error, and it inflates\nv\nmost in exactly the coordinates where the quantizer is struggling most. Adam responds by shrinking the step in those coordinates. So feedback pushes the correction into the numerator while simultaneously telling the optimizer to distrust the coordinates it is correcting. That is why the two effects compound into something worse than doing nothing, rather than merely cancelling to nothing.\nThe fix, and the ablation I should have run first\nThe standard repair is to freeze\nv\nafter a warmup period, which is what 1-bit Adam (Tang et al., 2021) does for compressed gradient communication. A frozen second moment is a fixed diagonal preconditioner, and a fixed preconditioner is a linear map, so error feedback's assumption is restored. I froze\nv\nat step 500 of 4,000, measured\n1.67e-5\n, wrote \"repairs it completely,\" and moved on.\nThat number should have stopped me. 1.67e-5 is not a repair to the full-precision level of 8.86e-5; it is\nfive times better\nthan the full-precision run. No fix to a quantization defect can beat not quantizing. Something else was in the measurement, and I published the claim without isolating it.\nSo here is the missing two-by-two. Two arms nobody had run: freeze\nv\nwith no quantization at all, and quantize with frozen\nv\nbut no feedback.\narm (all Adam)\ndistance to optimum\nfull precision\n8.86e-5\nfull precision,\nv\nfrozen\n2.04e-5\nquantized\n9.58e-5\nquantized,\nv\nfrozen, no feedback\n1.88e-5\nquantized + error feedback\n1.86e-4\nquantized + error feedback,\nv\nfrozen\n1.67e-5\nFreezing\nv\nis worth 4.3 times on this problem\nwith no quantization in the run at all\n. That, and not the repair of anything, is where the 1.67e-5 comes from: the frozen-\nv\narms all sit together between 1.67e-5 and 2.04e-5, quantized or not, corrected or not. And the two arms that separate feedback's contribution (1.88e-5 without it, 1.67e-5 with it) differ by 2.1e-6 against a three-sigma threshold of 4.5e-6, so on this problem, once\nv\nis frozen,\nerror feedback's measurable contribution is nothing at all.\nThe honest reading is narrower than what I published and slightly more interesting. Freezing\nv\ndoes not restore error feedback's validity so much as remove the conditions under which feedback was hurting. Feedback stops being actively harmful, and stops doing anything either. Meanwhile most of the headline gain is a different optimizer being better suited to a constant-learning-rate convex objective, which is not a result about quantization and will not necessarily survive contact with a non-stationary pretraining loss.\nThe correction to the earlier write-up:\nit reported this arm as error feedback being repaired, in a table whose other rows are fine. The landmine is unaffected; the repair row is confounded, and the deployable rule changes shape. Not \"error feedback needs frozen\nv\nunder Adam\" but \"gradient-space error feedback under vanilla Adam is harmful, and freezing\nv\nremoves the harm without demonstrating any benefit from the feedback.\"\nOne smaller correction while I'm here. The earlier post published 1.78e-4 for the Adam feedback arm, and re-running the same script twice for this post gave 1.86e-4 both times, against an across-seed spread of 2.4e-5. Every ordering is unchanged, but 1.86e-4 is the number I can reproduce and 1.78e-4 is the number I published.\nWhat this does not establish\nThe setting is coarse on purpose, and the caveats are proportional to that.\nThe quantizer's 28.2% relative MSE is about 40 times NVFP4's, so the\nmagnitude\nhere says nothing about the magnitude at four bits, where the harm should be proportionally smaller. That prediction has weak support from a rented B200 run in the same project: the feedback arm finished +0.01% against bf16, indistinguishable from noise. It was silently running the broken configuration and nothing bad visibly happened, which is not reassurance so much as an illustration that at FP4 the effect is below the floor in either direction.\nThe objective is convex least squares with a constant learning rate, which is where the frozen-\nv\ngain lives and probably where it dies. A fixed preconditioner reduces steady-state jitter on a stationary problem; a real pretraining run has a schedule, a loss landscape that keeps moving, and gradient statistics that shift under it. I would not carry the 4.3 times over, and the ablation cannot tell me what replaces it.\nWhat does travel is the ordering, which is a claim about algebra rather than about scale: under an optimizer that is nonlinear in the gradient, the correction can have the wrong sign, and nothing in the setup will warn you.\nWhat generalizes\nA correction inherits the algebra of whatever consumes it. Error feedback is correct, its proof is fine, and its assumption is stated plainly in the papers that introduce it; I simply carried it across a boundary where it stops holding, because it kept being described as a property of the compressor and it is actually a property of the compressor\nand\nthe optimizer together. My GPU runs had it bolted onto AdamW the whole time.\nThe part I'll actually remember is smaller and more embarrassing.\nWhen a fix outperforms the thing it was supposed to restore, the fix is doing something you haven't measured.\n1.67e-5 against a full-precision 8.86e-5 was five times too good, sitting in my own table, and I read it as a strong result instead of an anomaly. The ablation that explained it is two extra arms of an existing script and ran in a couple of minutes on a laptop. The expensive part was never the experiment; it was being pleased with the number.",
  "metadata": {
    "title": "Error Feedback, Gradient Compression, and Why Adam Breaks It - DEV Community",
    "description": "Error feedback makes a biased gradient compressor unbiased over time, and under SGD it restored the full-precision trajectory to three digits. Under Adam it landed 1.9 times further from the optimum than no correction at all, and the fix I published helps just as much with no quantization in the run. Tagged with llm, measurement, quantization, training.",
    "keywords": "llm, measurement, quantization, training, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/megapixel99/error-feedback-gradient-compression-and-why-adam-breaks-it-pm4",
    "og:title": "Error Feedback, Gradient Compression, and Why Adam Breaks It",
    "og:description": "Error feedback makes a biased gradient compressor unbiased over time, and under SGD it restored the full-precision trajectory to three digits. Under Adam it landed 1.9 times further from the optimum than no correction at all, and the fix I published helps just as much with no quantization in the run.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsyflhv8hta5iq60e5vlc.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Error Feedback, Gradient Compression, and Why Adam Breaks It",
    "twitter:description": "Error feedback makes a biased gradient compressor unbiased over time, and under SGD it restored the full-precision trajectory to three digits. Under Adam it landed 1.9 times further from the optimum than no correction at all, and the fix I published helps just as much with no quantization in the run.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsyflhv8hta5iq60e5vlc.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:41Z",
  "source_url": "https://dev.to/megapixel99/error-feedback-gradient-compression-and-why-adam-breaks-it-pm4",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 10859,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ae4879c46eba/pages/source.html",
  "local_html": "attachments/ae4879c46eba/pages/article.html",
  "body_base_path": "attachments/ae4879c46eba/pages/",
  "body_markdown_path": "attachments/ae4879c46eba/pages/article.md"
}