{
  "source_file": "0930",
  "source_platform": "Ghost",
  "author": "Avi Chawla / Daily Dose of Data Science",
  "title_listed": "5 LLM Quantization Techniques",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.dailydoseofds.com/p/5-llm-quantization-techniques/",
  "publish_date": "2026-08-07",
  "corpus_stage": "",
  "notes": "",
  "id": "5b1bdef030e7",
  "collected_at": "2026-10-02T13:46:20",
  "title": "5 LLM Quantization Techniques",
  "body_markdown": "Aug 8, 2026\nLLMOps\n5 LLM Quantization Techniques\n...explained visually.\nAvi Chawla\n👉\nA 70B parameter model in FP16 requires 140GB of memory for weights alone.\nThat exceeds any single GPU available today:\nAn H100 carries 80GB\nAn RTX 4090 carries 24GB\nSo serving the model in half precision means multi-GPU setups and tensor parallelism before the first token is generated.\nQuantization reduces this footprint by storing weights in lower bits.\nThe standard scheme maps each weight from FP16 onto a small integer grid: compute the dynamic range of the tensor, divide it into\n2^b\nlevels (16 levels at 4-bit), round each weight to the nearest level, and store a scale factor to dequantize during inference.\nAt 4-bit, the weight memory drops 4x. The 140GB model fits in 35GB, within the range of a single 40GB or 48GB card.\nThat said, if rounding were lossless in practice, one method would suffice, and the diagram below would have had one row:\nBut it has five because weights and activations in large transformers are not uniformly important.\nThe\nLLM.int8()\npaper quantified this. Beyond roughly 6.7B parameters, every transformer layer develops a handful of hidden dimensions carrying values 20x to 100x larger than the rest.\nThese dimensions make up about 0.1% of the model’s features, but zeroing them out nearly destroys the model’s ability to predict text.\nThese outliers are also what break naive rounding. The quantization grid spans the range of the tensor, so a single activation of magnitude 60 among values near 1 inflates the scale factor and collapses the remaining values into a few levels, destroying their precision.\nEach method in the visual shown earlier solves this saliency problem differently.\n1)\nRTN\n(round to nearest) does not handle it at all. It applies the scale-and-round procedure directly with no calibration data, which makes it the cheapest baseline and the weakest at low bit widths, where accumulated rounding error is unrecoverable.\n2)\nGPTQ\ncorrects rounding damage as it happens. It quantizes the weights of a layer a few at a time, measures how much error the rounding just introduced, and nudges the not-yet-quantized weights to cancel out that error before moving on.\nThe adjustment is not uniform. GPTQ uses statistics from calibration data to work out which remaining weights can absorb the error with the least effect on the layer’s output, which is what separates it from blind rounding.\nThis runs fast enough to matter in practice. A 175B model quantizes to 4-bit in around four GPU hours. The known weakness is that fitting the calibration data this closely can tune the model to it, and the AWQ authors showed this hurts accuracy on inputs that look different from the calibration set.\n3)\nAWQ\nprotects the important weights before rounding instead of repairing damage afterward. It runs calibration samples through the model and watches which weights get multiplied by the largest incoming values, because a rounding error on those weights gets amplified the most in the layer’s output.\nOnly around 1% of weight channels turn out to matter this much. Rather than storing them at higher precision, AWQ multiplies them by a scaling factor before rounding, so they spread across more levels of the grid and lose less detail, and the math is rearranged so a matching inverse scale elsewhere keeps the layer’s output unchanged.\nEvery weight still ends up in plain INT4, with no special-case handling at inference time. AWQ also depends far less on which calibration samples were used than GPTQ does, which is a big part of why it became the standard choice in serving engines like vLLM.\n4)\nLLM.int8()\nisolates the outliers instead of fighting them. When the model is loaded, it finds the dimensions carrying extreme values and splits every matrix multiplication into two:\nThe outliers run in full FP16\nThe other 99.9% of values run in INT8, and the two results are added back together.\nNothing needs to be computed ahead of time. That is why it is also available as the\nload_in_8bit\nflag in\nbitsandbytes\n.\nThe cost is speed. Splitting and merging every matrix multiplication is slower than running one optimized 4-bit kernel, so this path is common for local development and QLoRA fine-tuning, not for serving traffic at scale.\n5)\nQAT\nmodifies training rather than the quantization procedure. During a short fine-tuning run, every forward pass rounds the weights to INT4 before using them, so the model experiences the exact damage that quantization will cause, and its predictions are scored under that damage.\nThe weight updates still happen in full precision behind the scenes, since rounding itself cannot be trained through directly. Over the fine-tuning run, the weights drift toward values that keep the loss low even after rounding, so when the real quantization is applied at the end, there is almost nothing left to break.\nGoogle produced its Gemma 3 QAT checkpoints with roughly 5,000 fine-tuning steps guided by the original model’s outputs, and the int4 versions hold close to full-precision quality at about 3x lower memory.\nAll five methods described above produce the same artifact, i.e., a model at a fraction of its trained precision.\nThey differ in where the outlier problem gets addressed, at rounding time, after rounding, before rounding, at inference, or during training itself.\nTo dive deeper, we have already written a full\ndeep dive on Quantization\n, specifically, which covers several of these methods with their simplified mathematics.\nLearn how Quantization optimizes LLMs to run them on tiny hardware here →\nGood day!\nPublished on Aug 8, 2026\nComments\nShare\nCopy link\nShare to X\nShare to Facebook\nShare to Linkedin\nCopied\nPrevious\nWhy Your Agent Remembers Everything and Understands Nothing\nNext\nPreloading Knowledge Into a Model Instead of Retrieving It (Part A)",
  "metadata": {
    "title": "5 LLM Quantization Techniques",
    "description": "...explained visually.",
    "keywords": null,
    "author": null,
    "og:site_name": "Daily Dose of Data Science",
    "og:type": "article",
    "og:title": "5 LLM Quantization Techniques",
    "og:description": "...explained visually.",
    "og:url": "https://www.dailydoseofds.com/p/5-llm-quantization-techniques/",
    "og:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w1200/2026/08/5-LLM-quantization-techniques.jpg",
    "og:image:width": "1200",
    "og:image:height": "1067",
    "twitter:card": "summary_large_image",
    "twitter:title": "5 LLM Quantization Techniques",
    "twitter:description": "...explained visually.",
    "twitter:url": "https://www.dailydoseofds.com/p/5-llm-quantization-techniques/",
    "twitter:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w1200/2026/08/5-LLM-quantization-techniques.jpg",
    "twitter:label1": "Written by",
    "twitter:data1": "Avi Chawla",
    "twitter:label2": "Filed under",
    "twitter:data2": "LLMOps, LLMs",
    "twitter:site": "@_avichawla",
    "article:published_time": "2026-08-07T20:11:48.000Z",
    "article:modified_time": "2026-08-07T20:11:48.000Z",
    "article:tag": "LLMs"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "bytes": 11344,
      "reused": true,
      "local_path": "attachments/5b1bdef030e7/images/000.jpg",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "alt": "Avi Chawla",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!mcn3!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F437225d5-c382-4f8e-9ea3-88bb9b0e78b5_1554x274.gif",
      "bytes": 718560,
      "reused": true,
      "local_path": "attachments/5b1bdef030e7/images/001.gif",
      "src": "https://substackcdn.com/image/fetch/$s_!mcn3!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F437225d5-c382-4f8e-9ea3-88bb9b0e78b5_1554x274.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!BYVx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6090d643-2b7b-40be-a391-abd20cbf8d99_2899x991.png",
      "bytes": 89311,
      "reused": true,
      "local_path": "attachments/5b1bdef030e7/images/002.png",
      "src": "https://substackcdn.com/image/fetch/$s_!BYVx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6090d643-2b7b-40be-a391-abd20cbf8d99_2899x991.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!jdZa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82be21c1-7649-4d25-8e4b-b900dda507c0_1350x1200.jpeg",
      "bytes": 183068,
      "reused": true,
      "local_path": "attachments/5b1bdef030e7/images/003.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!jdZa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82be21c1-7649-4d25-8e4b-b900dda507c0_1350x1200.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!MCnt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb306f134-0557-4da8-a23d-ad979dd5306c_4707x1105.png",
      "bytes": 118271,
      "reused": true,
      "local_path": "attachments/5b1bdef030e7/images/004.png",
      "src": "https://substackcdn.com/image/fetch/$s_!MCnt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb306f134-0557-4da8-a23d-ad979dd5306c_4707x1105.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Cq7M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7d5a73e-26e0-4bf9-b48c-48b841ff5b65_1200x896.jpeg",
      "bytes": 161252,
      "reused": true,
      "local_path": "attachments/5b1bdef030e7/images/005.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!Cq7M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7d5a73e-26e0-4bf9-b48c-48b841ff5b65_1200x896.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!gLDQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1615326b-b912-4c5d-86f3-a89d727141dc_1338x216.png",
      "bytes": 65810,
      "reused": true,
      "local_path": "attachments/5b1bdef030e7/images/006.png",
      "src": "https://substackcdn.com/image/fetch/$s_!gLDQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1615326b-b912-4c5d-86f3-a89d727141dc_1338x216.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Ewpo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62adc36e-0ca4-4f04-a661-04f1ae9a1ef0_1338x214.png",
      "bytes": 86109,
      "reused": true,
      "local_path": "attachments/5b1bdef030e7/images/007.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Ewpo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62adc36e-0ca4-4f04-a661-04f1ae9a1ef0_1338x214.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!c6nX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8dcb2c2d-36cb-429d-b4ec-dc6d6e35837f_1338x214.png",
      "bytes": 68912,
      "reused": true,
      "local_path": "attachments/5b1bdef030e7/images/008.png",
      "src": "https://substackcdn.com/image/fetch/$s_!c6nX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8dcb2c2d-36cb-429d-b4ec-dc6d6e35837f_1338x214.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!tej0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F741c84cf-d58e-4d07-9700-50467aa67772_1338x214.png",
      "bytes": 82041,
      "reused": true,
      "local_path": "attachments/5b1bdef030e7/images/009.png",
      "src": "https://substackcdn.com/image/fetch/$s_!tej0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F741c84cf-d58e-4d07-9700-50467aa67772_1338x214.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!mfLX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22678dba-f5c8-458b-95c1-fb07f317ed48_1338x231.png",
      "bytes": 86770,
      "reused": true,
      "local_path": "attachments/5b1bdef030e7/images/010.png",
      "src": "https://substackcdn.com/image/fetch/$s_!mfLX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22678dba-f5c8-458b-95c1-fb07f317ed48_1338x231.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!fatV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c860c15-3e43-4403-9344-dccf0b0e3988_1004x442.png",
      "bytes": 41609,
      "reused": true,
      "local_path": "attachments/5b1bdef030e7/images/011.png",
      "src": "https://substackcdn.com/image/fetch/$s_!fatV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c860c15-3e43-4403-9344-dccf0b0e3988_1004x442.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.dailydoseofds.com/quantization-optimize-ml-models-to-run-them-on-tiny-hardware/",
      "label": "deep dive on Quantization",
      "type": "a"
    },
    {
      "href": "https://x.com/intent/tweet?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2F5-llm-quantization-techniques%2F&text=5%20LLM%20Quantization%20Techniques",
      "label": "Share to X",
      "type": "a"
    },
    {
      "href": "https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2F5-llm-quantization-techniques%2F",
      "label": "Share to Facebook",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2F5-llm-quantization-techniques%2F",
      "label": "Share to Linkedin",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/p/why-your-agent-remembers-everything-and-understands-nothing-2/",
      "label": "Previous Why Your Agent Remembers Everything and Understands Nothing",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/building-rag-systems-course-part-13-with-implementation/",
      "label": "Next Preloading Knowledge Into a Model Instead of Retrieving It (Part A)",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:55Z",
  "source_url": "https://www.dailydoseofds.com/p/5-llm-quantization-techniques/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 5829,
  "expected_body_images": 12,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/5b1bdef030e7/pages/source.html",
  "local_html": "attachments/5b1bdef030e7/pages/article.html",
  "body_base_path": "attachments/5b1bdef030e7/pages/",
  "body_markdown_path": "attachments/5b1bdef030e7/pages/article.md"
}