{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Maxime Labonne",
  "title_listed": "4-bit Quantization with GPTQ",
  "keywords": "量化,GPTQ,AutoGPTQ",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "部分免费/会员墙",
  "url": "https://towardsdatascience.com/4-bit-quantization-with-gptq-36b0f4f02c34/",
  "publish_date": "2023-07",
  "corpus_stage": "预训练/SFT",
  "notes": "",
  "id": "29847482acd9",
  "collected_at": "2026-10-01T21:53:55",
  "title": "4-bit Quantization with GPTQ | Towards Data Science",
  "body_markdown": "Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# 4-bit Quantization with GPTQ\nQuantize your own LLMs using AutoGPTQ\n[Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\nJuly 31, 202310 min read\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1Yq09FF2qH56PFg-LQsyhKQ.jpeg)\nImage by author\nRecent advancements in weight quantization allow us to run massive large language models on consumer hardware, like a LLaMA-30B model on an RTX 3090 GPU. This is possible thanks to novel 4-bit quantization techniques with minimal performance degradation, like [GPTQ](https://arxiv.org/abs/2210.17323), [GGML](https://github.com/ggerganov/ggml), and [NF4](https://huggingface.co/blog/4bit-transformers-bitsandbytes).\nIn the [previous article](https://medium.com/towards-data-science/introduction-to-weight-quantization-2494701b9c0c), we introduced naïve 8-bit quantization techniques and the excellent LLM.int8(). In this article, we will explore the popular **GPTQ algorithm** to understand how it works and implement it using the [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) library.\nYou can find the code on [Google Colab](https://colab.research.google.com/drive/1lSvVDaRgqQp_mWK_jC9gydz6_-y6Aq4A?usp=sharing) and [GitHub](https://github.com/mlabonne/llm-course/tree/main).\n## 🧠 Optimal Brain Quantization\nLet's start by introducing the problem we're trying to solve. For every layer ℓ in the network, we want to find a quantized version **Ŵₗ** _of the original weights_ **Wₗ**. This is called the **layer-wise compression problem**. More specifically, to minimize performance degradation, we want the outputs (**Ŵ** ᵨ**X** ᵨ) of these new weights to be as close as possible to the original ones (**W** ᵨ**X** ᵨ). In other words, we want to find:\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer), [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/102PIN1yLtRRKiQlAJbg5ZQ.png)\nDifferent approaches have been proposed to solve this problem, but we're interested in the [**Optimal Brain Quantizer**](https://arxiv.org/abs/2208.11580) (OBQ) framework here.\nThis method is inspired by a **pruning technique** to carefully remove weights from a fully trained dense neural network (Optimal Brain Surgeon). It uses an approximation technique and provides explicit formulas for the best single weight _w𐞥_ to remove and optimal update δ _ꟳ_ to adjust the set of remaining non-quantized weights F __ to make up for the removal:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1wBuyKG70KOydcIC2Sk7-0g.png)\nwhere quant(_w_) is the weight rounding given by the quantization and **H** ꟳ is the Hessian.\nUsing OBQ, we can quantize the easiest weight first and then adjust all remaining non-quantized weights to **compensate for this precision loss**. Then we pick the next weight to quantize, and so on.\nA potential issue with this approach is when there are outlier weights, which can result in high **quantization error**. Usually, these outliers would be quantized last, when there are few non-quantized weights left that could be adjusted to compensate for the large error. This effect can worsen when some weights are pushed further outside the grid by intermediate updates. A simple heuristic is applied to prevent this: outliers are quantized as soon as they appear.\nThis process could be computationally heavy, especially for LLMs. To deal with this, the OBQ method uses a trick that avoids redoing the entire computation each time a weight is simplified. After quantizing a weight, it adjusts the matrix used in calculations (the Hessian) by **removing the row and column** associated with that weight (using Gaussian elimination):\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1QOJT761baVB9BJb4R3XlBA.png)\nThe method also employs vectorization to process multiple rows of the weight matrix at once. Despite its efficiency, the OBQ's computation time increases significantly as the size of the weight matrix increases. This cubic growth makes it difficult to use OBQ on very large models with billions of parameters.\n## 🧮 The GPTQ Algorithm\nIntroduced by Frantar et al. (2023), the [GPTQ algorithm](https://arxiv.org/abs/2210.17323) takes inspiration from the OBQ method, but with significant improvements to scale it for (very) large language models.\n### Step 1: Arbitrary Order Insight\nThe OBQ method selects weights (parameters in a model) for quantization in a certain order, determined by which will **add the least additional error**. However, GPTQ observes that for large models, quantizing weights in any fixed order can perform just as well. This is because even though some weights might introduce more error individually, they are quantized later in the process when there are few other weights left that could increase the error. So the order doesn't matter as much as we thought.\nBased on this insight, GPTQ aims to quantize all weights in the **same order for all rows** of a matrix. This makes the process faster because certain computations have to be done only once for each column, rather than once for each weight.\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1x9fTHoe83Enkd2spdla3oQ.png)\nImage by author\n### Step 2: Lazy Batch-Updates\nThis scheme won't be fast because it requires updating a **huge matrix** with very few computations for each entry. This type of operation can't utilize the full compute capabilities of GPUs and will be slowed down by memory limitations (memory throughput bottleneck).\nTo resolve this, GPTQ introduces \"lazy batch\" updates. It turns out that the final rounding decisions for a given column are only affected by updates performed on that column, not on later columns. Therefore, GPTQ can apply the algorithm to a **batch of columns at a time** (like 128 columns), updating only those columns and a corresponding block of the matrix. After a block is fully processed, the algorithm performs global updates on the entire matrix.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1B_4K-L0Sq05f1HX5nxNKyA.png)\n### Step 3: Cholesky Reformulation\nHowever, there's one more issue to address. When the algorithm scales up to very large models, numerical inaccuracies can become a problem. Specifically, repeated applications of a certain operation can **accumulate numerical errors**.\nTo tackle this, GPTQ uses a [Cholesky decomposition](https://en.wikipedia.org/wiki/Cholesky_decomposition), a numerically stable method for solving certain mathematical problems. It involves precomputing some required information from the matrix using the Cholesky method. This approach, combined with a slight \"dampening\" (adding a small constant to diagonal elements of the matrix), helps the algorithm to avoid numerical issues.\nThe full algorithm can be summarized in a few steps:\n  1. The GPTQ algorithm begins with a Cholesky decomposition of the Hessian inverse (a matrix that helps decide how to adjust the weights)\n  2. It then runs in loops, handling batches of columns at a time.\n  3. For each column in a batch, it quantizes the weights, calculates the error, and updates the weights in the block accordingly.\n  4. After processing the batch, it updates all remaining weights based on the block's errors.\n\n\nThe GPTQ algorithm was tested on various language generation tasks. It was compared with other quantization methods, like rounding all weights to the nearest quantized value (RTN). GPTQ was used with the BLOOM (176B parameters) and OPT (175B parameters) model families, and models were quantized using a **single NVIDIA A100 GPU**.\n## 💻 Quantize an LLM with AutoGPTQ\nGPTQ has been very popular to create models in 4-bit precision that can efficiently run on GPUs. You can find many examples on the Hugging Face Hub, especially from [TheBloke](https://huggingface.co/TheBloke). If you're looking for an approach that is more CPU-friendly, [GGML](https://github.com/ggerganov/ggml) is currently your best option. Finally, the `transformers` library with `bitsandbytes` allows you to quantize a model when it's loaded using the `load_in_4bit=true` argument, which requires downloading full models and storing them in your RAM.\nLet's implement the GPTQ algorithm using the AutoGPTQ library and quantize a GPT-2 model. This requires a GPU, but a free T4 on Google Colab will do. We start by loading the libraries and defining the model we want to quantize (in this case, GPT-2).\nbash\n\n```\n!BUILD_CUDA_EXT=0 pip install -q auto-gptq transformers\n```\n\npython\n\n```\nimport randomfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfigfrom datasets import load_datasetimport torchfrom transformers import AutoTokenizer# Define base model and output directorymodel_id = \"gpt2\"out_dir = model_id + \"-GPTQ\"\n```\n\nWe now want to load the model and the tokenizer. The tokenizer is loaded using the classic `AutoTokenizer` class from the `transformers` library. On the other hand, we need to pass a specific configuration (`BaseQuantizeConfig`) to load the model.\nIn this configuration, we can specify the number of bits to quantize (here, `bits=4`) and the group size (size of the lazy batch). Note that this group size is optional: we could also use **one set of parameters** for the entire weight matrix. In practice, these groups generally improve the quality of the quantization at a very low cost (especially with `group_size=1024`). The `damp_percent` value is here to help the Cholesky reformulation and should not be changed.\nFinally, the `desc_act` (also called act order) is a tricky parameter. It allows you to **process rows based on decreasing activation** , meaning the most important or impactful rows (determined by sampled inputs and outputs) are processed first. This method aims to place most of the quantization error (inevitably introduced during quantization) on less significant weights. This approach improves the overall accuracy of the quantization process by ensuring the most significant weights are processed with greater precision. However, when used alongside group size, `desc_act` can lead to performance slowdowns due to the need to frequently reload quantization parameters. For this reason, we won't use it here (it will probably be fixed in the future, however).\npython\n\n```\n# Load quantize config, model and tokenizerquantize_config = BaseQuantizeConfig(    bits=4,    group_size=128,    damp_percent=0.01,    desc_act=False,)model = AutoGPTQForCausalLM.from_pretrained(model_id, quantize_config)tokenizer = AutoTokenizer.from_pretrained(model_id)\n```\n\nThe quantization process **relies heavily on samples** to evaluate and enhance the quality of the quantization. They provide a means of comparison between the outputs produced by the origina and the newly quantized model. The larger the number of samples provided, the greater the potential for more accurate and effective comparisons, leading to improved quantization quality.\nIn the context of this article, we utilize the [**C4 (Colossal Clean Crawled Corpus) dataset**](https://huggingface.co/datasets/c4) to generate our samples. The C4 dataset is a large-scale, multilingual collection of web text gathered from the Common Crawl project. This expansive dataset has been cleaned and prepared specifically for training large-scale language models, making it a great resource for tasks such as this. The WikiText dataset is another popular option.\nIn the following code block, we load 1024 samples from the C4 dataset, tokenize them, and format them.\npython\n\n```\n# Load data and tokenize examplesn_samples = 1024data = load_dataset(\"allenai/c4\", data_files=\"en/c4-train.00001-of-01024.json.gz\", split=f\"train[:{n_samples*5}]\")tokenized_data = tokenizer(\"nn\".join(data['text']), return_tensors='pt')# Format tokenized examplesexamples_ids = []for _ in range(n_samples):    i = random.randint(0, tokenized_data.input_ids.shape[1] - tokenizer.model_max_length - 1)    j = i + tokenizer.model_max_length    input_ids = tokenized_data.input_ids[:, i:j]    attention_mask = torch.ones_like(input_ids)    examples_ids.append({'input_ids': input_ids, 'attention_mask': attention_mask})\n```\n\nNow that dataset is ready, we can start the quantization process with a batch size of 1. Optionally, we also use [OpenAI Triton](https://github.com/openai/triton), a CUDA alternative, to communicate with the GPU. Once this is done, we save the tokenizer and the model in a safetensors format.\npython\n\n```\n# Quantize with GPTQmodel.quantize(    examples_ids,    batch_size=1,    use_triton=True,)# Save model and tokenizermodel.save_quantized(out_dir, use_safetensors=True)tokenizer.save_pretrained(out_dir)\n```\n\nAs per usual, the model and tokenizer can then be loaded from the output directory using the `AutoGPTQForCausalLM` and `AutoTokenizer` classes.\npython\n\n```\ndevice = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"# Reload model and tokenizermodel = AutoGPTQForCausalLM.from_quantized(    out_dir,    device=device,    use_triton=True,    use_safetensors=True,)tokenizer = AutoTokenizer.from_pretrained(out_dir)\n```\n\nLet's check that the model is working correctly. The AutoGPTQ model (mostly) works as a normal `transformers` model, which makes it compatible with inference pipelines, as shown in the following example:\npython\n\n```\nfrom transformers import pipelinegenerator = pipeline('text-generation', model=model, tokenizer=tokenizer)result = generator(\"I have a dream\", do_sample=True, max_length=50)[0]['generated_text']print(result)\n```\n\nplaintext\n\n```\nI have a dream,\" she told CNN last week. \"I have this dream of helping my mother find her own. But, to tell that for the first time, now that I'm seeing my mother now, just knowing how wonderful it is that\n```\n\nWe managed to get a convincing completion from our quantized GPT-2 model. A more in-depth evaluation would require **measuring the perplexity** of the quantized model versus the original one. However, we will leave it out of the scope of this article.\n## Conclusion\nIn this article, we introduced the GPTQ algorithm, a state-of-the-art quantization technique to run LLMs on consumer-grade hardware. We showed how it addresses the layer-wise compression problem, based on an improved OBS technique with arbitrary order insight, lazy batch updates, and Cholesky reformulation. This novel approach **significantly reduces memory and computation requirements** , making LLMs accessible to a broader audience.\nIn addition, we **quantized our own LLM model** on a free T4 GPU and ran it to generate text. You can push your own version of a GPTQ 4-bit quantized model on the Hugging Face Hub. As mentioned in the introduction, GPTQ is not the only 4-bit quantization algorithm: [GGML](https://github.com/ggerganov/ggml) and [NF4](https://huggingface.co/blog/4bit-transformers-bitsandbytes) are excellent alternatives with slightly different scopes. I encourage you to learn more about them and give them a shot!\nIf you're interested in more technical content around LLMs, follow me on Twitter [@maximelabonne](https://twitter.com/maximelabonne).\n## References\n  * B. Hassibi, D. G. Stork and G. J. Wolff, [\"Optimal Brain Surgeon and general network pruning,\"](https://ieeexplore.ieee.org/document/298572) IEEE International Conference on Neural Networks, San Francisco, CA, USA, 1993, pp. 293–299 vol.1, doi: 10.1109/ICNN.1993.298572.\n  * Elias Frantar, Sidak Pal Singh, & Dan Alistarh. (2023). [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://arxiv.org/abs/2208.11580).\n  * Elias Frantar, Saleh Ashkboos, Torsten Hoefler, & Dan Alistarh. (2023). [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323).\n  * Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, & Peter J. Liu. (2020). [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683v3).\n\n\n## Related articles\n> [**Introduction to Weight Quantization**](https://towardsdatascience.com/introduction-to-weight-quantization-2494701b9c0c)\n> [**Fine-Tune Your Own Llama 2 Model in a Colab Notebook**](https://towardsdatascience.com/fine-tune-your-own-llama-2-model-in-a-colab-notebook-df9823a04a32)\n* * *\n_Learn more about machine learning and support my work with one click - become a Medium member here:_\n> [**Join Medium with my referral link - Maxime Labonne**](https://medium.com/@mlabonne/membership)\n_If you're already a member, you can_[ _follow me on Medium_](https://medium.com/@mlabonne) _._\n* * *\n![](https://assets.insightmediagroup.io/media/1787657499317_sxy3ci.png)\nWritten By\n## [Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[See all from Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Data Science](https://towardsdatascience.com/tag/data-science/)[Large Language Models](https://towardsdatascience.com/tag/large-language-models/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2F4-bit-quantization-with-gptq-36b0f4f02c34%2F&title=4-bit%20Quantization%20with%20GPTQ)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2F4-bit-quantization-with-gptq-36b0f4f02c34%2F&title=4-bit%20Quantization%20with%20GPTQ)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2F4-bit-quantization-with-gptq-36b0f4f02c34%2F&text=4-bit%20Quantization%20with%20GPTQ)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/09/1vuEf1IntgEpKEtjZo030Ig.png)](https://towardsdatascience.com/deep-learning-how-to-build-an-emotional-chatbot-part-2-the-dialogue-system-4932afe6545c/)\n## [[Deep learning] How to build an emotional chatbot (Part 2 the dialogue system)](https://towardsdatascience.com/deep-learning-how-to-build-an-emotional-chatbot-part-2-the-dialogue-system-4932afe6545c/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nFrom part 1, we have built the BERT sentiment predictor, we now start combining it and building the emotional dialogue system. Before jump...\n[Jay Hui](https://towardsdatascience.com/author/jay-hui/)\nSeptember 18, 20207 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/00HYKLD3NmEmWdhzZ-scaled.jpg)](https://towardsdatascience.com/were-hiring-ad69101cdfc1/)\n## [We're Hiring!](https://towardsdatascience.com/were-hiring-ad69101cdfc1/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nTDS is looking for a Canada-based Senior Editor. Maybe that's you?\n[TDS Editors](https://towardsdatascience.com/author/towardsdatascience/)\nDecember 29, 20212 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/09/1x-68Wq-C5MiRZgsH7r3z2Q.gif)](https://towardsdatascience.com/how-to-calculate-travel-time-for-any-location-in-the-world-56ce639511f/)\n## [How to calculate Travel time for any location in the world](https://towardsdatascience.com/how-to-calculate-travel-time-for-any-location-in-the-world-56ce639511f/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nA guide on how to perform Isochrones (travel time from one point to another) in Python.\n[Abdishakur](https://towardsdatascience.com/author/shakasom/)\nSeptember 26, 20195 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/1ItZnim8bgUg2kd0NfqRKJQ-scaled.jpeg)](https://towardsdatascience.com/active-and-upcoming-reinforcement-learning-competitions-8ed3c4fb14ab/)\n## [7+ Reinforcement learning competitions to check out in 2022](https://towardsdatascience.com/active-and-upcoming-reinforcement-learning-competitions-8ed3c4fb14ab/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA list of ongoing and annual competitions suitable for getting hands-on practice with reinforcement learning (while winning a prize or...\n[Joy Zhang](https://towardsdatascience.com/author/joooyz/)\nOctober 28, 20214 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/04/pawel-czerwinski-0xCCPIbl3M-unsplash.jpg)](https://towardsdatascience.com/ai-in-social-research-and-polling/)\n## [AI in Social Research and Polling](https://towardsdatascience.com/ai-in-social-research-and-polling/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nWhat do we do when nobody answers the phone?\n[Stephanie Kirmer](https://towardsdatascience.com/author/s.kirmer/)\nApril 1, 202511 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0nJYWP3a4iND_5rX2-scaled.jpg)](https://towardsdatascience.com/pandas-series-dataframe-explained-a178f9748d46/)\n## [Pandas Series & DataFrame Explained](https://towardsdatascience.com/pandas-series-dataframe-explained-a178f9748d46/)\n[Programming](https://towardsdatascience.com/category/programming/)\nA comprehensive guide to understanding Pandas Series and DataFrame data structures\n[Dean McGrath](https://towardsdatascience.com/author/deanmcgrath/)\nOctober 9, 20204 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/1M9WYxL2RMMnUSxNQ_hmFJA.png)](https://towardsdatascience.com/language-invariant-properties-in-natural-language-processing-561e04b5fa0d/)\n## [Language Invariant Properties in Natural Language Processing](https://towardsdatascience.com/language-invariant-properties-in-natural-language-processing-561e04b5fa0d/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nAre text properties kept when you transform (e.g., translate) a message in another language?\n[Federico Bianchi](https://towardsdatascience.com/author/fede-bianchi/)\nOctober 13, 20216 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/02/0zBQ89p4dqPfCngKE-scaled.jpg)](https://towardsdatascience.com/how-to-solve-travelling-salesman-problem-with-simulated-annealing-c248447a8bcd/)\n## [How To Solve Travelling Salesman Problem With Simulated Annealing](https://towardsdatascience.com/how-to-solve-travelling-salesman-problem-with-simulated-annealing-c248447a8bcd/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nGetting the optimal solution to the Travelling Salesman Problem using the Simulated Annealing optimisation algorithm\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nFebruary 10, 20232 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/02/image-175.jpg)](https://towardsdatascience.com/generalizable-marl-lp-approach-for-scheduling-in-logistics/)\n## [A Generalizable MARL-LP Approach for Scheduling in Logistics](https://towardsdatascience.com/generalizable-marl-lp-approach-for-scheduling-in-logistics/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nPart 1. Hybrid Solution for Dynamic Vehicle Routing — Context and Architecture\n[Alexander Levin](https://towardsdatascience.com/author/alexander.levin/)\nFebruary 26, 202614 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n",
  "metadata": {
    "title": "4-bit Quantization with GPTQ | Towards Data Science",
    "description": "Quantize your own LLMs using AutoGPTQ",
    "keywords": null,
    "author": "Maxime Labonne",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "4-bit Quantization with GPTQ",
    "og:description": "Quantize your own LLMs using AutoGPTQ",
    "og:url": "https://towardsdatascience.com/4-bit-quantization-with-gptq-36b0f4f02c34/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1Yq09FF2qH56PFg-LQsyhKQ.jpeg",
    "og:image:width": "1820",
    "og:image:height": "1024",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Maxime Labonne",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "10 min read",
    "article:published_time": "2023-07-31T13:52:52.000Z",
    "article:modified_time": "2023-07-31T13:52:52.000Z",
    "article:author": "https://towardsdatascience.com/author/mlabonne/"
  },
  "paywall": {
    "listed_paid": true,
    "detected": false,
    "marker": "",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/29847482acd9/code/block_000.txt",
      "chars": 56
    },
    {
      "language": "txt",
      "file": "attachments/29847482acd9/code/block_001.txt",
      "chars": 243
    },
    {
      "language": "txt",
      "file": "attachments/29847482acd9/code/block_002.txt",
      "chars": 274
    },
    {
      "language": "txt",
      "file": "attachments/29847482acd9/code/block_003.txt",
      "chars": 615
    },
    {
      "language": "txt",
      "file": "attachments/29847482acd9/code/block_004.txt",
      "chars": 202
    },
    {
      "language": "txt",
      "file": "attachments/29847482acd9/code/block_005.txt",
      "chars": 255
    },
    {
      "language": "txt",
      "file": "attachments/29847482acd9/code/block_006.txt",
      "chars": 208
    },
    {
      "language": "txt",
      "file": "attachments/29847482acd9/code/block_007.txt",
      "chars": 206
    }
  ],
  "images": [
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1Yq09FF2qH56PFg-LQsyhKQ.jpeg",
      "alt": "Image by author",
      "local_path": "attachments/29847482acd9/images/000.jpeg",
      "bytes": 58451,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1QOJT761baVB9BJb4R3XlBA.png",
      "alt": "",
      "local_path": "attachments/29847482acd9/images/001.png",
      "bytes": 18353,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1x9fTHoe83Enkd2spdla3oQ.png",
      "alt": "Image by author",
      "local_path": "attachments/29847482acd9/images/002.png",
      "bytes": 50969,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/09/1vuEf1IntgEpKEtjZo030Ig.png",
      "alt": "",
      "local_path": "attachments/29847482acd9/images/003.png",
      "bytes": 75814,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/00HYKLD3NmEmWdhzZ-scaled.jpg",
      "alt": "",
      "local_path": "attachments/29847482acd9/images/004.jpg",
      "bytes": 695663,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/09/1x-68Wq-C5MiRZgsH7r3z2Q.gif",
      "alt": "",
      "local_path": "attachments/29847482acd9/images/005.gif",
      "bytes": 563611,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/1ItZnim8bgUg2kd0NfqRKJQ-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/29847482acd9/images/006.jpeg",
      "bytes": 314316,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/04/pawel-czerwinski-0xCCPIbl3M-unsplash.jpg",
      "alt": "",
      "local_path": "attachments/29847482acd9/images/007.jpg",
      "bytes": 201330,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0nJYWP3a4iND_5rX2-scaled.jpg",
      "alt": "",
      "local_path": "attachments/29847482acd9/images/008.jpg",
      "bytes": 1049464,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/1M9WYxL2RMMnUSxNQ_hmFJA.png",
      "alt": "",
      "local_path": "attachments/29847482acd9/images/009.png",
      "bytes": 50031,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/02/0zBQ89p4dqPfCngKE-scaled.jpg",
      "alt": "",
      "local_path": "attachments/29847482acd9/images/010.jpg",
      "bytes": 392081,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/02/image-175.jpg",
      "alt": "",
      "local_path": "attachments/29847482acd9/images/011.jpg",
      "bytes": 206759,
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "src": "https://www.youtube.com/c/TowardsDataScience",
      "quality": "unavailable",
      "local_path": "",
      "note": "无1080p且无720p，未交付更低清晰度"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/mlabonne/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/introduction-to-weight-quantization-2494701b9c0c"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/fine-tune-your-own-llama-2-model-in-a-colab-notebook-df9823a04a32"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/large-language-models/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/deep-learning-how-to-build-an-emotional-chatbot-part-2-the-dialogue-system-4932afe6545c/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/deep-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/jay-hui/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/were-hiring-ad69101cdfc1/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/towardsdatascience/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2210.17323"
    },
    {
      "type": "external",
      "href": "https://github.com/ggerganov/ggml"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/blog/4bit-transformers-bitsandbytes"
    },
    {
      "type": "external",
      "href": "https://medium.com/towards-data-science/introduction-to-weight-quantization-2494701b9c0c"
    },
    {
      "type": "external",
      "href": "https://github.com/PanQiWei/AutoGPTQ"
    },
    {
      "type": "external",
      "href": "https://colab.research.google.com/drive/1lSvVDaRgqQp_mWK_jC9gydz6_-y6Aq4A?usp=sharing"
    },
    {
      "type": "external",
      "href": "https://github.com/mlabonne/llm-course/tree/main"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-engineer"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-data-scientist"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/machine-learning"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2208.11580"
    },
    {
      "type": "external",
      "href": "https://en.wikipedia.org/wiki/Cholesky_decomposition"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/TheBloke"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/datasets/c4"
    },
    {
      "type": "external",
      "href": "https://github.com/openai/triton"
    },
    {
      "type": "external",
      "href": "https://twitter.com/maximelabonne"
    },
    {
      "type": "external",
      "href": "https://ieeexplore.ieee.org/document/298572"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T11:11:48Z",
  "source_url": "https://towardsdatascience.com/4-bit-quantization-with-gptq-36b0f4f02c34/",
  "delivery_status": "excluded_paid",
  "discovery_status": "excluded_paid",
  "missing_reason": "按原约定排除付费/会员全文"
}