{ "source_file": "0930", "source_platform": "Medium", "author": "Maxime Labonne", "title_listed": "Introduction to Weight Quantization", "keywords": "量化,8-bit,INT8", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "部分免费/会员墙", "url": "https://towardsdatascience.com/introduction-to-weight-quantization-2494701b9c0c/", "publish_date": "2023-07", "corpus_stage": "预训练/SFT", "notes": "", "id": "59e1b5ff7b1d", "collected_at": "2026-10-01T21:53:55", "title": "Introduction to Weight Quantization | Towards Data Science", "body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Science](https://towardsdatascience.com/category/data-science/)\n# Introduction to Weight Quantization\nReducing the size of Large Language Models with 8-bit quantization\n[Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\nJuly 7, 202314 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1hWIaIAQ7GWbrjfbaoUoYxw.jpeg)\nLarge Language Models (LLMs) are known for their extensive computational requirements. Typically, the size of a model is calculated by multiplying the number of parameters (**size**) by the precision of these values (**data type**). However, to save memory, weights can be stored using lower-precision data types through a process known as quantization.\nWe distinguish two main families of weight quantization techniques in the literature:\n * **Post-Training Quantization** (PTQ) is a straightforward technique where the weights of an already trained model are converted to lower precision without necessitating any retraining. Although easy to implement, PTQ is associated with potential performance degradation.\n * **Quantization-Aware Training** (QAT) incorporates the weight conversion process during the pre-training or fine-tuning stage, resulting in enhanced model performance. However, QAT is computationally expensive and demands representative training data.\n\n\nIn this article, we focus on PTQ to reduce the precision of our parameters. To get a good intuition, we will apply both naïve and more sophisticated techniques to a toy example using a GPT-2 model.\nThe entire code is freely available on [Google Colab](https://colab.research.google.com/drive/1DPr4mUQ92Cc-xf4GgAaB6dFcFnWIvqYi?usp=sharing) and [GitHub](https://github.com/mlabonne/llm-course/blob/main/Introduction_to_Weight_Quantization.ipynb).\nLearn this step by step with the interactive [AI and Data Scientist](https://roadmap.sh/ai-data-scientist), [AI Engineer](https://roadmap.sh/ai-engineer) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps.\n## 📚 Background on Floating Point Representation\nThe choice of data type dictates the quantity of computational resources required, affecting the speed and efficiency of the model. In deep learning applications, balancing precision and computational performance becomes a vital exercise as higher precision often implies greater computational demands.\nAmong various data types, floating point numbers are predominantly employed in deep learning due to their ability to represent a wide range of values with high precision. Typically, a floating point number uses _n_ bits to store a numerical value. These _n_ bits are further partitioned into three distinct components:\n 1. **Sign** : The sign bit indicates the positive or negative nature of the number. It uses one bit where 0 indicates a positive number and 1 signals a negative number.\n 2. **Exponent** : The exponent is a segment of bits that represents the power to which the base (usually 2 in binary representation) is raised. The exponent can also be positive or negative, allowing the number to represent very large or very small values.\n 3. **Significand/Mantissa** : The remaining bits are used to store the significand, also referred to as the mantissa. This represents the significant digits of the number. The precision of the number heavily depends on the length of the significand.\n\n\nThis design allows floating point numbers to cover a wide range of values with varying levels of precision. The formula used for this representation is:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1gVngsuChFURCIf0N9biYWw.png)\nTo understand this better, let's delve into some of the most commonly used data types in deep learning: float32 (FP32), float16 (FP16), and bfloat16 (BF16):\n * **FP32** uses 32 bits to represent a number: one bit for the sign, eight for the exponent, and the remaining 23 for the significand. While it provides a high degree of precision, the downside of FP32 is its high computational and memory footprint.\n * **FP16** uses 16 bits to store a number: one is used for the sign, five for the exponent, and ten for the significand. Although this makes it more memory-efficient and accelerates computations, the reduced range and precision can introduce numerical instability, potentially impacting model accuracy.\n * **BF16** is also a 16-bit format but with one bit for the sign, _eight_ for the exponent, and _seven_ for the significand. BF16 expands the representable range compared to FP16, thus decreasing underflow and overflow risks. Despite a reduction in precision due to fewer significand bits, BF16 typically does not significantly impact model performance and is a useful compromise for deep learning tasks.\n\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1maBUGGMedWvvDyk6FdvkWw.png)\nImage by author\nIn ML jargon, FP32 is often termed \"full precision\" (4 bytes), while BF16 and FP16 are \"half-precision\" (2 bytes). But could we do even better and store weights using a single byte? The answer is the INT8 data type, which consists of an 8-bit representation capable of storing 2⁸ = 256 different values. In the next section, we'll see how to convert FP32 weights into an INT8 format.\n## 🔰 Naïve 8-bit Quantization\nIn this section, we will implement two quantization techniques: a symmetric one with **absolute maximum (absmax) quantization** and an asymmetric one with **zero-point quantization**. In both cases, the goal is to map an FP32 tensor **X** (original weights) to an INT8 tensor **X_quant** (quantized weights).\nWith **absmax quantization** , the original number is divided by the absolute maximum value of the tensor and multiplied by a scaling factor (127) to map inputs into the range [-127, 127]. To retrieve the original FP16 values, the INT8 number is divided by the quantization factor, acknowledging some loss of precision due to rounding.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1jNl_x4JF0lpRA4_6Cae9cg.png)\nFor instance, let's say we have an absolution maximum value of 3.2. A weight of 0.1 would be quantized to _round(0.1 × 127/3.2) = 4_. If we want to dequantize it, we would get _4 × 3.2/127 = 0.1008_ , which implies an error of 0.008. Here's the corresponding Python implementation:\npython\n\n```\nimport torch\n\ndef absmax_quantize(X):\n # Calculate scale\n scale = 127 / torch.max(torch.abs(X))\n\n # Quantize\n X_quant = (scale * X).round()\n\n # Dequantize\n X_dequant = X_quant / scale\n\n return X_quant.to(torch.int8), X_dequant\n```\n\nWith **zero-point quantization** , we can consider asymmetric input distributions, which is useful when you consider the output of a ReLU function (only positive values), for example. The input values are first scaled by the total range of values (255) divided by the difference between the maximum and minimum values. This distribution is then shifted by the zero-point to map it into the range [-128, 127] (notice the extra value compared to absmax). First, we calculate the scale factor and the zero-point value:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1GiCuvGWBtdU4-hXcvetTnw.png)\nThen, we can use these variables to quantize or dequantize our weights:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1SalTtt_eNoYOeHLD1XiEfw.png)\nLet's take an example: we have a maximum value of 3.2 and a minimum value of -3.0. We can calculate the scale is _255/(3.2 + 3.0) = 41.13_ and the zero-point _-round(41.13 × -3.0) - 128 = 123 -128 = -5_ , so our previous weight of 0.1 would be quantized to _round(41.13 × 0.1 -5) = -1_. This is very different from the previous value obtained using absmax (4 vs. -1).\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1n5nqoJUXp65JahKsLzQS-A.png)\nImage by author\nThe Python implementation is quite straightforward:\npython\n\n```\ndef zeropoint_quantize(X):\n # Calculate value range (denominator)\n x_range = torch.max(X) - torch.min(X)\n x_range = 1 if x_range == 0 else x_range\n\n # Calculate scale\n scale = 255 / x_range\n\n # Shift by zero-point\n zeropoint = (-scale * torch.min(X) - 128).round()\n\n # Scale and round the inputs\n X_quant = torch.clip((X * scale + zeropoint).round(), -128, 127)\n\n # Dequantize\n X_dequant = (X_quant - zeropoint) / scale\n\n return X_quant.to(torch.int8), X_dequant\n```\n\nInstead of relying on complete toy examples, we can use these two functions on a real model thanks to the `transformers`library.\nWe start by loading the model and tokenizer for GPT-2. This is a very small model we probably don't want to quantize, but it will be good enough for this tutorial. First, we want to observe the model's size so we can compare it later and evaluate the **memory savings** due to 8-bit quantization.\nbash\n\n```\n!pip install -q bitsandbytes>=0.39.0\n!pip install -q git+https://github.com/huggingface/accelerate.git\n!pip install -q git+https://github.com/huggingface/transformers.git\n```\n\npython\n\n```\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\ntorch.manual_seed(0)\n\n# Set device to CPU for now\ndevice = 'cpu'\n\n# Load model and tokenizer\nmodel_id = 'gpt2'\nmodel = AutoModelForCausalLM.from_pretrained(model_id).to(device)\ntokenizer = AutoTokenizer.from_pretrained(model_id)\n\n# Print model size\nprint(f\"Model size: {model.get_memory_footprint():,} bytes\")\n```\n\nmarkdown\n\n```\nModel size: 510,342,192 bytes\n```\n\nThe size of the GPT-2 model is approximately 487MB in FP32. The next step consists of quantizing the weights using zero-point and absmax quantization. In the following example, we apply these techniques to the first attention layer of GPT-2 to see the results.\npython\n\n```\n# Extract weights of the first layer\nweights = model.transformer.h[0].attn.c_attn.weight.data\nprint(\"Original weights:\")\nprint(weights)\n\n# Quantize layer using absmax quantization\nweights_abs_quant, _ = absmax_quantize(weights)\nprint(\"nAbsmax quantized weights:\")\nprint(weights_abs_quant)\n\n# Quantize layer using absmax quantization\nweights_zp_quant, _ = zeropoint_quantize(weights)\nprint(\"nZero-point quantized weights:\")\nprint(weights_zp_quant)\n```\n\nmarkdown\n\n```\nOriginal weights:\ntensor([[-0.4738, -0.2614, -0.0978, ..., 0.0513, -0.0584, 0.0250],\n [ 0.0874, 0.1473, 0.2387, ..., -0.0525, -0.0113, -0.0156],\n [ 0.0039, 0.0695, 0.3668, ..., 0.1143, 0.0363, -0.0318],\n ...,\n [-0.2592, -0.0164, 0.1991, ..., 0.0095, -0.0516, 0.0319],\n [ 0.1517, 0.2170, 0.1043, ..., 0.0293, -0.0429, -0.0475],\n [-0.4100, -0.1924, -0.2400, ..., -0.0046, 0.0070, 0.0198]])\n\nAbsmax quantized weights:\ntensor([[-21, -12, -4, ..., 2, -3, 1],\n [ 4, 7, 11, ..., -2, -1, -1],\n [ 0, 3, 16, ..., 5, 2, -1],\n ...,\n [-12, -1, 9, ..., 0, -2, 1],\n [ 7, 10, 5, ..., 1, -2, -2],\n [-18, -9, -11, ..., 0, 0, 1]], dtype=torch.int8)\n\nZero-point quantized weights:\ntensor([[-20, -11, -3, ..., 3, -2, 2],\n [ 5, 8, 12, ..., -1, 0, 0],\n [ 1, 4, 18, ..., 6, 3, 0],\n ...,\n [-11, 0, 10, ..., 1, -1, 2],\n [ 8, 11, 6, ..., 2, -1, -1],\n [-18, -8, -10, ..., 1, 1, 2]], dtype=torch.int8)\n```\n\nThe difference between the original (FP32) and quantized values (INT8) is clear, but the difference between absmax and zero-point weights is more subtle. In this case, the inputs look shifted by a value of -1. This suggests that the weight distribution in this layer is quite symmetric.\nWe can compare these techniques by quantizing every layer in GPT-2 (linear layers, attention layers, etc.) and create two new models: `model_abs` and `model_zp`. To be precise, we will actually replace the original weights with _**de**_ -quantized ones. This has two benefits: it allows us to 1/ compare the distribution of our weights (same scale) and 2/ actually run the models.\nIndeed, PyTorch doesn't allow INT8 matrix multiplication by default. In a real scenario, we would dequantize them to run the model (in FP16 for example) but store them as INT8. In the next section, we will use the `[bitsandbytes](https://github.com/TimDettmers/bitsandbytes)` library to solve this issue.\npython\n\n```\nimport numpy as np\nfrom copy import deepcopy\n\n# Store original weights\nweights = [param.data.clone() for param in model.parameters()]\n\n# Create model to quantize\nmodel_abs = deepcopy(model)\n\n# Quantize all model weights\nweights_abs = []\nfor param in model_abs.parameters():\n _, dequantized = absmax_quantize(param.data)\n param.data = dequantized\n weights_abs.append(dequantized)\n\n# Create model to quantize\nmodel_zp = deepcopy(model)\n\n# Quantize all model weights\nweights_zp = []\nfor param in model_zp.parameters():\n _, dequantized = zeropoint_quantize(param.data)\n param.data = dequantized\n weights_zp.append(dequantized)\n```\n\nNow that our models have been quantized, we want to check the impact of this process. Intuitively, we want to make sure that the quantized weights are **close to the original ones**. A visual way to check it is to plot the distribution of the dequantized and original weights. If the quantization is lossy, it would drastically change the weight distribution.\nThe following figure shows this comparison, where the blue histogram represents the original (FP32) weights, and the red one represents the dequantized (from INT8) weights. Note that we only display this plot between -2 and 2 because of outliers with very high absolute values (more on that later).\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/15FkPW5PQDn3zqP7x9QSPLg.png)\nBoth plots are quite similar, with a surprising spike around 0. This spike shows that our quantization is quite lossy since reversing the process doesn't output the original values. This is particularly true for the absmax model, which displays both a lower valley and a higher spike around 0.\nLet's compare the performance of the original and quantized models. For this purpose, we define a `generate_text()` function to generate 50 tokens with [top-k sampling](https://mlabonne.github.io/blog/posts/2023-06-07-Decoding_strategies.html).\npython\n\n```\ndef generate_text(model, input_text, max_length=50):\n input_ids = tokenizer.encode(input_text, return_tensors='pt').to(device)\n output = model.generate(inputs=input_ids,\n max_length=max_length,\n do_sample=True,\n top_k=30,\n pad_token_id=tokenizer.eos_token_id,\n attention_mask=input_ids.new_ones(input_ids.shape))\n return tokenizer.decode(output[0], skip_special_tokens=True)\n\n# Generate text with original and quantized models\noriginal_text = generate_text(model, \"I have a dream\")\nabsmax_text = generate_text(model_abs, \"I have a dream\")\nzp_text = generate_text(model_zp, \"I have a dream\")\n\nprint(f\"Original model:n{original_text}\")\nprint(\"-\" * 50)\nprint(f\"Absmax model:n{absmax_text}\")\nprint(\"-\" * 50)\nprint(f\"Zeropoint model:n{zp_text}\")\n```\n\nmarkdown\n\n```\nOriginal model:\nI have a dream, and it is a dream I believe I would get to live in my future. I love my mother, and there was that one time I had been told that my family wasn't even that strong. And then I got the\n--------------------------------------------------\nAbsmax model:\nI have a dream to find out the origin of her hair. She loves it. But there's no way you could be honest about how her hair is made. She must be crazy.\n\nWe found a photo of the hairstyle posted on\n--------------------------------------------------\nZeropoint model:\nI have a dream of creating two full-time jobs in America—one for people with mental health issues, and one for people who do not suffer from mental illness—or at least have an employment and family history of substance abuse, to work part\n```\n\nInstead of trying to see if one output makes more sense than the others, we can quantify it by calculating the **perplexity** of each output. This is a common metric used to evaluate language models, which measures the uncertainty of a model in predicting the next token in a sequence. In this comparison, we make the common assumption that the lower the score, the better the model is. In practice, a sentence with a high perplexity could also be correct.\nWe implement it using a minimal function since it doesn't need to consider details like the length of the context window since our sentences are short.\npython\n\n```\ndef calculate_perplexity(model, text):\n # Encode the text\n encodings = tokenizer(text, return_tensors='pt').to(device)\n\n # Define input_ids and target_ids\n input_ids = encodings.input_ids\n target_ids = input_ids.clone()\n\n with torch.no_grad():\n outputs = model(input_ids, labels=target_ids)\n\n # Loss calculation\n neg_log_likelihood = outputs.loss\n\n # Perplexity calculation\n ppl = torch.exp(neg_log_likelihood)\n\n return ppl\n\nppl = calculate_perplexity(model, original_text)\nppl_abs = calculate_perplexity(model_abs, absmax_text)\nppl_zp = calculate_perplexity(model_zp, absmax_text)\n\nprint(f\"Original perplexity: {ppl.item():.2f}\")\nprint(f\"Absmax perplexity: {ppl_abs.item():.2f}\")\nprint(f\"Zeropoint perplexity: {ppl_zp.item():.2f}\")\n```\n\nmarkdown\n\n```\nOriginal perplexity: 15.53\nAbsmax perplexity: 17.92\nZeropoint perplexity: 17.97\n```\n\nWe see that the perplexity of the original model is **slightly lower** than the two others. A single experiment is not very reliable, but we could repeat this process multiple times to see the difference between each model. In theory, zero-point quantization should be slightly better than absmax, but is also more costly to compute.\nIn this example, we applied quantization techniques to entire layers (per-tensor basis). However, we could apply it at different granularity levels: from the entire model to individual values. Quantizing the entire model in one pass would seriously degrade the performance, while quantizing individual values would create a big overhead. In practice, we often prefer the **vector-wise quantization** , which considers the variability of values in rows and columns inside of the same tensor.\nHowever, even vector-wise quantization doesn't solve the problem of outlier features. Outlier features are extreme values (negative or positive) that appear in all transformer layers when the model reach a certain scale (>6.7B parameters). This is an issue since a single outlier can reduce the precision for all other values. But discarding these outlier features is not an option since it would **greatly degrade** the model's performance.\n## 🔢 8-bit Quantization with LLM.int8()\nIntroduced by [Dettmers et al. (2022)](https://arxiv.org/abs/2208.07339), LLM.int8() is a solution to the outlier problem. It relies on a vector-wise (absmax) quantization scheme and introduces mixed-precision quantization. This means that outlier features are processed in a FP16 format to retain their precision, while the other values are processed in an INT8 format. As outliers represent about 0.1% of values, this effectively reduces the memory footprint of the LLM by almost 2x.\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1SFxEfA8009WcgWJG5Vsh0A.png)\nImage by author\nLLM.int8() works by conducting matrix multiplication computation in three key steps:\n 1. Extract columns from the input hidden states **X** containing outlier features using a custom threshold.\n 2. Perform the matrix multiplication of the outliers using FP16 and the non-outliers using INT8 with vector-wise quantization (row-wise for the hidden state **X** and column-wise for the weight matrix **W**).\n 3. Dequantize the non-outlier results (INT8 to FP16) and add them to the outlier results to get the full result in FP16.\n\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/11xt5F9kFLFGfiFmRtF11Pg.png)\nImage by author\nThis approach is necessary because 8-bit precision is limited and can lead to substantial errors when quantizing a vector with large values. These errors also tend to amplify as they propagate through multiple layers.\nWe can easily use this technique thanks to the integration of the `bitsandbytes` library into the Hugging Face ecosystem. We just need to specify `load_in_8bit=True` when loading the model (it also requires a GPU).\npython\n\n```\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n\nmodel_int8 = AutoModelForCausalLM.from_pretrained(model_id,\n device_map='auto',\n load_in_8bit=True,\n )\nprint(f\"Model size: {model_int8.get_memory_footprint():,} bytes\")\n```\n\nmarkdown\n\n```\nModel size: 176,527,896 bytes\n```\n\nWith this extra line of code, the model is now almost three times smaller (168MB vs. 487MB). We can even compare the distribution of the original and quantized weights as we did earlier:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1gvBKaY8nDJgTfMwPVi77KQ.png)\nIn this case, we see spikes around -2, -1, 0, 1, 2, etc. These values correspond to the parameters stored in the INT8 format (non-outliers). You can verify it by printing the model's weights using `model_int8.parameters()`.\nWe can also generate text with this quantized model and compare it to the original model.\npython\n\n```\n# Generate text with quantized model\ntext_int8 = generate_text(model_int8, \"I have a dream\")\n\nprint(f\"Original model:n{original_text}\")\nprint(\"-\" * 50)\nprint(f\"LLM.int8() model:n{text_int8}\")\n```\n\nmarkdown\n\n```\nOriginal model:\nI have a dream, and it is a dream I believe I would get to live in my future. I love my mother, and there was that one time I had been told that my family wasn't even that strong. And then I got the\n--------------------------------------------------\nLLM.int8() model:\nI have a dream. I don't know what will come of it, but I am going to have to look for something that will be right. I haven't thought about it for a long time, but I have to try to get that thing\n```\n\nOnce again, it is difficult to judge what is the best output, but we can rely on the perplexity metric to give us an (approximate) answer.\npython\n\n```\nprint(f\"Perplexity (original): {ppl.item():.2f}\")\n\nppl = calculate_perplexity(model_int8, text_int8)\nprint(f\"Perplexity (LLM.int8()): {ppl.item():.2f}\")\n```\n\nmarkdown\n\n```\nPerplexity (original): 15.53\nPerplexity (LLM.int8()): 7.93\n```\n\nIn this case, the perplexity of the quantized model is twice as low as the original one. In general, this is not the case, but it shows that this quantization technique is very competitive. In fact, the authors of LLM.int8() show that the performance degradation is so low it's negligible (<1%). However, it has an additional cost in terms of computation: LLM.int8() is roughly about 20% slower for large models.\n## Conclusion\nThis article provided an overview of the most popular weight quantization techniques. We started by gaining an understanding of floating point representation, before introducing two techniques for 8-bit quantization: **absmax** and **zero-point quantization**. However, their limitations, particularly when it comes to handling outliers, led to **LLM.int8()** , a technique that also preserves the model's performance. This approach underlines the progress being made in the field of weight quantization, revealing the importance of properly addressing outliers.\nLooking forward, our next article will explore the GPTQ weight quantization technique in depth. This technique, introduced by [Frantar et al.](https://arxiv.org/abs/2210.17323), only utilizes 4 bits and represents a significant advancement in the field of weight quantization. We will provide a comprehensive guide on how to implement GPTQ using the AutoGPTQ library.\nIf you're interested in more technical content around LLMs, follow me on Twitter [@maximelabonne](https://twitter.com/maximelabonne).\n## References\n * T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339). 2022.\n * Y. Beldaka, and T. Dettmers, [A Gentle Introduction to 8-bit Matrix Multiplication](https://huggingface.co/blog/hf-bitsandbytes-integration), Hugging Face Blog (2022).\n * A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, [A Survey of Quantization Methods for Efficient Neural Network Inference](https://arxiv.org/abs/2103.13630). 2021.\n * H. Wu, P. Judd, X. Zhang, M. Isaev, and P. Micikevicius, [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://arxiv.org/abs/2004.09602). 2020.\n * Lilian Weng, [Large Transformer Model Inference Optimization](https://lilianweng.github.io/posts/2023-01-10-inference-optimization/), Lil'Log (2023).\n * Kamil Czarnogorski, [Local Large Language Models](https://int8.io/local-large-language-models-beginners-guide/), Int8 (2023).\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657499317_sxy3ci.png)\nWritten By\n## [Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[See all from Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[Data Science](https://towardsdatascience.com/tag/data-science/)[Hands On Tutorials](https://towardsdatascience.com/tag/hands-on-tutorials/)[Large Language Models](https://towardsdatascience.com/tag/large-language-models/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Quantization](https://towardsdatascience.com/tag/quantization/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fintroduction-to-weight-quantization-2494701b9c0c%2F&title=Introduction%20to%20Weight%20Quantization)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fintroduction-to-weight-quantization-2494701b9c0c%2F&title=Introduction%20to%20Weight%20Quantization)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fintroduction-to-weight-quantization-2494701b9c0c%2F&text=Introduction%20to%20Weight%20Quantization)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1RKjnDQ6T7Me-xeYHuydSwA-scaled.jpeg)](https://towardsdatascience.com/my-experience-as-a-product-data-analyst-3d01748bc6ea/)\n## [My Experience as a Product Data Analyst](https://towardsdatascience.com/my-experience-as-a-product-data-analyst-3d01748bc6ea/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nHow supporting product made me a better data analyst\n[Vicky Yu](https://towardsdatascience.com/author/madfordata/)\nMarch 8, 20215 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1bgrd578BFL_snu6sGgypfA.png)](https://towardsdatascience.com/distribution-analyser-b826b88b7b8d/)\n## [Distribution Analyser](https://towardsdatascience.com/distribution-analyser-b826b88b7b8d/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nWeb application made with Streamlit\n[Robert Dzudzar](https://towardsdatascience.com/author/robertdzudzar/)\nApril 8, 20216 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/19JWQTzQXXa83a3mSK7VEcw.png)](https://towardsdatascience.com/kaggle-predict-survival-on-the-titanic-challenge-in-matlab-56f6ad3bab78/)\n## [Kaggle - Predict survival on the Titanic challenge in MATLAB](https://towardsdatascience.com/kaggle-predict-survival-on-the-titanic-challenge-in-matlab-56f6ad3bab78/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nIn this tutorial, I will demonstrate how to use MATLAB to solve this popular Machine Learning challenge on Kaggle.\n[Abhishek Das](https://towardsdatascience.com/author/iamabhishekdas/)\nJanuary 13, 20215 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/03/0PWZNzgCnBjpwq1EZ-scaled.jpg)](https://towardsdatascience.com/statistical-t-test-simply-explained-b510045d69e/)\n## [Statistical T-Test Simply Explained](https://towardsdatascience.com/statistical-t-test-simply-explained-b510045d69e/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nAn introduction to the Student's t-distribution and the Student's t-test\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nMarch 14, 20224 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/Data-team-scaled.jpeg)](https://towardsdatascience.com/platform-mesh-hub-and-spoke-and-centralised-3-types-of-data-team/)\n## [Platform-Mesh, Hub and Spoke, and Centralised | 3 Types of data team](https://towardsdatascience.com/platform-mesh-hub-and-spoke-and-centralised-3-types-of-data-team/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nWhy understanding team structure is critical for data and AI\n[Hugo Lu](https://towardsdatascience.com/author/hugolu87/)\nMarch 10, 20259 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/05/07w1R1OpO1ZYoPQBR-scaled.jpg)](https://towardsdatascience.com/how-to-use-gpt-j-for-almost-any-nlp-task-cb3ca8ff5826/)\n## [How to Use GPT-J for (Almost) Any NLP Task](https://towardsdatascience.com/how-to-use-gpt-j-for-almost-any-nlp-task-cb3ca8ff5826/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nPrompt engineering and how it can be used with text generation models\n[Heiko Hotz](https://towardsdatascience.com/author/heiko-hotz/)\nMay 6, 20225 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/05/1TOq4NWpaBm9GBRAlH0M5vQ.png)](https://towardsdatascience.com/a-better-way-to-analyze-feature-release-impact-72529f907ccd/)\n## [A better way to analyze feature release impact](https://towardsdatascience.com/a-better-way-to-analyze-feature-release-impact-72529f907ccd/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nOr - why naive \"before-after\" comparisons can drive bad product decisions\n[Iyar Lin](https://towardsdatascience.com/author/iyarlin/)\nMay 22, 20234 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1grfoP_RqFP0cwQnBCl-xSw@2x.png)](https://towardsdatascience.com/utilizing-pyarrow-to-improve-pandas-and-dask-workflows-2891d3d96d2b/)\n## [Utilizing PyArrow to improve pandas and Dask workflows](https://towardsdatascience.com/utilizing-pyarrow-to-improve-pandas-and-dask-workflows-2891d3d96d2b/)\n[Programming](https://towardsdatascience.com/category/programming/)\nGet the most out of PyArrow support in pandas and Dask right now\n[Patrick Hoefler](https://towardsdatascience.com/author/patrick_hoefler/)\nJune 6, 202314 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/0eNYg2THLMU6-8TAm-scaled.jpg)](https://towardsdatascience.com/get-going-with-neo4j-and-jupyter-lab-through-docker-a1994e0e95c6/)\n## [Get going with Neo4j and Jupyter Lab through Docker](https://towardsdatascience.com/get-going-with-neo4j-and-jupyter-lab-through-docker-a1994e0e95c6/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nWhy work with one at a time when you can work with both?\n[CJ Sullivan](https://towardsdatascience.com/author/cj2001/)\nMay 14, 20215 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "metadata": { "title": "Introduction to Weight Quantization | Towards Data Science", "description": "Reducing the size of Large Language Models with 8-bit quantization", "keywords": null, "author": "Maxime Labonne", "og:locale": "en_US", "og:type": "article", "og:title": "Introduction to Weight Quantization", "og:description": "Reducing the size of Large Language Models with 8-bit quantization", "og:url": "https://towardsdatascience.com/introduction-to-weight-quantization-2494701b9c0c/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1hWIaIAQ7GWbrjfbaoUoYxw.jpeg", "og:image:width": "1820", "og:image:height": "1024", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Maxime Labonne", "twitter:label2": "Est. reading time", "twitter:data2": "14 min read", "article:published_time": "2023-07-07T06:58:09.000Z", "article:modified_time": "2023-07-07T06:58:09.000Z", "article:author": "https://towardsdatascience.com/author/mlabonne/" }, "paywall": { "listed_paid": true, "detected": false, "marker": "", "public_only": true, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_000.txt", "chars": 248 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_001.txt", "chars": 499 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_002.txt", "chars": 171 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_003.txt", "chars": 384 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_004.txt", "chars": 30 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_005.txt", "chars": 447 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_006.txt", "chars": 1126 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_007.txt", "chars": 641 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_008.txt", "chars": 893 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_009.txt", "chars": 783 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_010.txt", "chars": 782 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_011.txt", "chars": 84 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_012.txt", "chars": 372 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_013.txt", "chars": 30 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_014.txt", "chars": 192 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_015.txt", "chars": 480 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_016.txt", "chars": 155 }, { "language": "txt", "file": "attachments/59e1b5ff7b1d/code/block_017.txt", "chars": 61 } ], "images": [ { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1hWIaIAQ7GWbrjfbaoUoYxw.jpeg", "alt": "", "local_path": "attachments/59e1b5ff7b1d/images/000.jpeg", "bytes": 71392, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1gVngsuChFURCIf0N9biYWw.png", "alt": "", "local_path": "attachments/59e1b5ff7b1d/images/001.png", "bytes": 24022, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1maBUGGMedWvvDyk6FdvkWw.png", "alt": "Image by author", "local_path": "attachments/59e1b5ff7b1d/images/002.png", "bytes": 437528, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1n5nqoJUXp65JahKsLzQS-A.png", "alt": "Image by author", "local_path": "attachments/59e1b5ff7b1d/images/003.png", "bytes": 405926, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1SFxEfA8009WcgWJG5Vsh0A.png", "alt": "Image by author", "local_path": "attachments/59e1b5ff7b1d/images/004.png", "bytes": 159234, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/11xt5F9kFLFGfiFmRtF11Pg.png", "alt": "Image by author", "local_path": "attachments/59e1b5ff7b1d/images/005.png", "bytes": 473332, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1RKjnDQ6T7Me-xeYHuydSwA-scaled.jpeg", "alt": "", "local_path": "attachments/59e1b5ff7b1d/images/006.jpeg", "bytes": 375437, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1bgrd578BFL_snu6sGgypfA.png", "alt": "", "local_path": "attachments/59e1b5ff7b1d/images/007.png", "bytes": 192663, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/19JWQTzQXXa83a3mSK7VEcw.png", "alt": "", "local_path": "attachments/59e1b5ff7b1d/images/008.png", "bytes": 9293346, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/03/0PWZNzgCnBjpwq1EZ-scaled.jpg", "alt": "", "local_path": "attachments/59e1b5ff7b1d/images/009.jpg", "bytes": 234433, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/Data-team-scaled.jpeg", "alt": "", "local_path": "attachments/59e1b5ff7b1d/images/010.jpeg", "bytes": 103392, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/05/07w1R1OpO1ZYoPQBR-scaled.jpg", "alt": "", "local_path": "attachments/59e1b5ff7b1d/images/011.jpg", "bytes": 409442, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/05/1TOq4NWpaBm9GBRAlH0M5vQ.png", "alt": "", "local_path": "attachments/59e1b5ff7b1d/images/012.png", "bytes": 1277641, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1grfoP_RqFP0cwQnBCl-xSw@2x.png", "alt": "", "local_path": "attachments/59e1b5ff7b1d/images/013.png", "bytes": 34576, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/0eNYg2THLMU6-8TAm-scaled.jpg", "alt": "", "local_path": "attachments/59e1b5ff7b1d/images/014.jpg", "bytes": 579798, "downloaded": true } ], "audio": [], "videos": [ { "src": "https://www.youtube.com/c/TowardsDataScience", "quality": "unavailable", "local_path": "", "note": "无1080p且无720p，未交付更低清晰度" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/data-science/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/mlabonne/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/data-science/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/hands-on-tutorials/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/large-language-models/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/quantization/" }, { "type": "internal", "href": "https://towardsdatascience.com/my-experience-as-a-product-data-analyst-3d01748bc6ea/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/madfordata/" }, { "type": "internal", "href": "https://towardsdatascience.com/distribution-analyser-b826b88b7b8d/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/robertdzudzar/" }, { "type": "internal", "href": "https://towardsdatascience.com/kaggle-predict-survival-on-the-titanic-challenge-in-matlab-56f6ad3bab78/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/iamabhishekdas/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://colab.research.google.com/drive/1DPr4mUQ92Cc-xf4GgAaB6dFcFnWIvqYi?usp=sharing" }, { "type": "external", "href": "https://github.com/mlabonne/llm-course/blob/main/Introduction_to_Weight_Quantization.ipynb" }, { "type": "external", "href": "https://roadmap.sh/ai-data-scientist" }, { "type": "external", "href": "https://roadmap.sh/ai-engineer" }, { "type": "external", "href": "https://roadmap.sh/machine-learning" }, { "type": "external", "href": "https://mlabonne.github.io/blog/posts/2023-06-07-Decoding_strategies.html" }, { "type": "external", "href": "https://arxiv.org/abs/2208.07339" }, { "type": "external", "href": "https://arxiv.org/abs/2210.17323" }, { "type": "external", "href": "https://twitter.com/maximelabonne" }, { "type": "external", "href": "https://huggingface.co/blog/hf-bitsandbytes-integration" }, { "type": "external", "href": "https://arxiv.org/abs/2103.13630" }, { "type": "external", "href": "https://arxiv.org/abs/2004.09602" }, { "type": "external", "href": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/" }, { "type": "external", "href": "https://int8.io/local-large-language-models-beginners-guide/" }, { "type": "external", "href": "https://www.facebook.com/sharer/sharer.php?title=Introduction+to+Weight+Quantization&u=https%3A%2F%2Ftowardsdatascience.com%2Fintroduction-to-weight-quantization-2494701b9c0c%2F" }, { "type": "external", "href": "https://www.linkedin.com/shareArticle?mini=true&title=Introduction+to+Weight+Quantization&url=https%3A%2F%2Ftowardsdatascience.com%2Fintroduction-to-weight-quantization-2494701b9c0c%2F" }, { "type": "external", "href": "https://x.com/share?text=Introduction+to+Weight+Quantization&url=https%3A%2F%2Ftowardsdatascience.com%2Fintroduction-to-weight-quantization-2494701b9c0c%2F" } ], "repair_version": 2, "repaired_at": "2026-10-14T11:11:48Z", "source_url": "https://towardsdatascience.com/introduction-to-weight-quantization-2494701b9c0c/", "delivery_status": "excluded_paid", "discovery_status": "excluded_paid", "missing_reason": "按原约定排除付费/会员全文" }