{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Shawhin Talebi",
  "title_listed": "Compressing Large Language Models (LLMs)",
  "keywords": "模型压缩,量化,剪枝",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "部分免费/会员墙",
  "url": "https://towardsdatascience.com/compressing-large-language-models-llms-9f406eea5b5e/",
  "publish_date": "2024-08",
  "corpus_stage": "预训练/SFT",
  "notes": "",
  "id": "1164e5b8f6f3",
  "collected_at": "2026-10-01T22:36:23",
  "title": "Compressing Large Language Models (LLMs) | Towards Data Science",
  "body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\n# Compressing Large Language Models (LLMs)\nMake LLMs 10X smaller without sacrificing performance\n[Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\nAugust 30, 202411 min read\n![Image from Canva.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1VDHOHFG-PA8VddO-tuTFbA.png)\nImage from Canva.\nThis article is part of a [larger series](https://shawhin.medium.com/list/large-language-models-llms-8e009ae3054c) on using large language models (LLMs) in practice. While the immense scale of LLMs is responsible for their impressive performance across a wide range of use cases, this presents **challenges in their application to real-world problems**. In this article, I discuss how we can overcome these challenges by compressing LLMs. I start with a high-level overview of key concepts and then walk through a concrete example with Python code.\n* * *\nPlease accept cookies to access this content\nThe AI mantra of 2023 was _\"Bigger is Better_ ,\" where the equation for improving language models was pretty simple: **more data + more parameters + more compute = better performance** [1].\nWhile this is likely still the case (GPT-5 coming soon?), there are obvious challenges with working with 100B+ parameter models. For example, a 100B parameter model using FP16 requires 200GB _just_ for storage!\nNeedless to say, most consumer devices (e.g. phones, tablets, laptops) can't handle models this big. _**But.. what if we could make them smaller?**_\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps.\n## **Model Compression**\n**Model compression** aims to **reduce the size of machine learning models without sacrificing performance** [2]. This works for (big) neural networks because they are often _over-parameterized_ (i.e. consist of redundant computational units) [3].\nThe key benefit of model compression is **lower inference costs**. This means wider accessibility of powerful ML models (i.e. running LLMs locally on your laptop), lower-cost integration of AI into consumer products, and on-device inference, which supports user privacy and security [3].\n## **3 Ways to Compress Models**\nThere is a wide range of techniques for model compression. Here, I will focus on 3 broad categories.\n  * **Quantization** - Representing models with lower precision data types\n  * **Pruning** - Removing unnecessary components from a model\n  * **Knowledge Distillation** - Training a smaller model using a bigger one\n\n\n_Note_ : these approaches are independent of one another. Thus, **techniques from multiple categories can be combined** for maximum compression!\n## **1) Quantization**\nWhile quantization might sound like a scary and sophisticated word, it is a simple idea. It consists of **lowering the precision of model parameters**. You can think of this as converting a high-resolution image to a lower-resolution one while still maintaining the picture's core properties.\n![Quantization analogy. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1M_0ZN9vERSCayZID56g1UA.png)\nQuantization analogy. Image by author.\nTwo common classes of quantization techniques are **Post-training Quantization (PTQ)** and **Quantization-Aware Training (QAT)**.\n### **1.1) Post-training Quantization (PTQ)**\nGiven a neural network, **Post-training Quantization (PTQ)** compresses the model by **replacing parameters with a lower-precision data type** (e.g. FP16 to INT-8). This is one of the fastest and simplest ways to reduce a model's computational requirements because it **requires no additional training or data labeling** [4].\nWhile this is a relatively easy way to cut model costs, excessive quantization in this way (e.g., FP16 to INT4) often leads to performance degradation, which limits the potential gains of PTQ. [3]\n### **1.2) Quantization-Aware Training (QAT)**\nFor situations where greater compression is needed, PTQ's limitations can be overcome by **training models (from scratch) with lower-precision data types**. This is the idea behind Quantization-Aware Training (QAT) [5].\nWhile this approach is more technically demanding, it can lead to a significantly smaller, well-performing model. For instance, the BitNet architecture used a **ternary data type (i.e. 1.58-bit)** to match the performance of the original Llama LLM [6]!\nOf course, a large technical gap exists between PTQ and from-scratch QAT. An approach between the two is **Quantization-aware Fine-tuning** , which consists of additional training of a pre-trained model after quantization [3].\n> [**QLoRA - How to Fine-Tune an LLM on a Single GPU**](https://towardsdatascience.com/qlora-how-to-fine-tune-an-llm-on-a-single-gpu-4e44d6b5be32)\n## **2) Pruning**\nThe goal of **pruning** is to **remove model components that have little impact on performance** [7]. This is effective because ML models (especially large ones) tend to learn redundant and noisy structures [3].\nAn analogy here is like clipping dead branches from a tree. Removing them reduces the size of the tree without harming it.\n![Pruning analogy. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1l9mhfjOIRnohnycafFXLIw.png)\nPruning analogy. Image by author.\nPruning approaches can be categorized into two buckets: **Unstructured** and **Structured Pruning**.\n### **2.1) Unstructured Pruning**\n**Unstructured pruning** **removes unimportant weights from a neural network** (i.e. setting them to zero). For example, early works such as Optimal Brain Damage and Optimal Brain Surgeon computed a saliency score for each parameter in the network by estimating the impact pruning on the loss function [7].\nMore recently, magnitude-based approaches (i.e. removing weights with the smallest absolute value) have become more popular due to their simplicity and scalability [7].\nWhile the granularity of unstructured pruning can significantly decrease parameter count, these **gains require specialized hardware to be realized** [7]. Unstructured pruning results in sparse matrix operations (i.e. multiplying matrixes with lots of zeros), which standard hardware cannot do more efficiently than non-sparse operations.\n### **2.2) Structured Pruning**\nAlternatively, **structured pruning** **removes entire structures from a neural network** (e.g. attention heads, neurons, and layers) [5]. This avoids the spare matrix operation problem because entire matrices can be dropped from the model rather than individual parameters.\nWhile there are various ways to identify structures for pruning, in principle, they all seek to remove structures with the smallest impact on performance. A survey of structured pruning approaches is available in reference [5].\n## **3) Knowledge Distillation**\n**Knowledge Distillation** **transfers knowledge from a (larger) teacher model to a (smaller) student model** [5]. One way to do this is to generate predictions with a teacher model and use them to train a student model. Learning from the teacher model's output logits (i.e., probabilities for all possible next tokens) provides richer information than the original training data, which improves student model performance [8].\nMore recent distillation applications **discard the need for logits** altogether and **learn from synthetic data generated from the teacher model**. A popular example is Stanford's Alpaca model, which fine-tuned the LLaMa 7B (foundation) model using synthetic data from OpenAI's text-davinci-003 (i.e. the original ChatGPT model), enabling it to follow user instructions [9].\n## **Example code: Compressing a Text Classifier with Knowledge Distillation + Quantization**\nWith a basic understanding of various compression techniques, let's see a hands-on example of how to do this in Python. Here, we will compress a 100M parameter model that classifies URLs as safe or unsafe (i.e. phishing).\nWe first use knowledge distillation to compress the 100M parameter model into a 50M parameter one. Then, using 4-bit quantization, we further reduced the memory footprint by 3X, resulting in a **final model that is 7X smaller than our original one**.\nThe example code is available on [GitHub](https://github.com/ShawhinT/YouTube-Blog/tree/main/LLMs/model-compression). The models ([Teacher](https://huggingface.co/shawhin/bert-phishing-classifier_teacher), [Student](https://huggingface.co/shawhin/bert-phishing-classifier_student), [Student-4bit](https://huggingface.co/shawhin/bert-phishing-classifier_student_4bit)) and [dataset](https://huggingface.co/datasets/shawhin/phishing-site-classification) are freely available on the Hugging Face Hub.\n* * *\nWe start by importing a few helpful libraries.\npython\n\n```\nfrom datasets import load_dataset\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\nfrom transformers import DistilBertForSequenceClassification, DistilBertConfig\n\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import DataLoader\n\nfrom sklearn.metrics import accuracy_score, precision_recall_fscore_support\n```\n\nThen, we load our dataset from the Hugging Face Hub. This includes training (2100 rows), testing (450 rows), and validation (450 rows) sets.\npython\n\n```\ndata = load_dataset(\"shawhin/phishing-site-classification\")\n```\n\nNext, we load in our teacher model. To help speed up training, I loaded the model onto a T4 GPU that was freely available on Google Colab.\npython\n\n```\n# use Nvidia GPU\ndevice = torch.device('cuda')\n\n# Load teacher model and tokenizer\nmodel_path = \"shawhin/bert-phishing-classifier_teacher\"\n\ntokenizer = AutoTokenizer.from_pretrained(model_path)\nteacher_model = AutoModelForSequenceClassification.from_pretrained(model_path)\n                                                  .to(device)\n```\n\nThe teacher model is a fine-tuned version of Goolge's [bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased) that performs binary classification on phishing website URLs. The code to train the teacher model is available on [GitHub](https://github.com/ShawhinT/YouTube-Blog/tree/main/LLMs/model-compression).\nFor the student model, we initialize a new model from scratch based on [distilbert-base-uncased](https://huggingface.co/distilbert/distilbert-base-uncased). We modify the architecture by removing two layers and four attention heads from the remaining layers.\npython\n\n```\n# Load student model\nmy_config = DistilBertConfig(n_heads=8, n_layers=4) # drop 4 heads per layer and 2 layers\n\nstudent_model = DistilBertForSequenceClassification\n                                    .from_pretrained(\"distilbert-base-uncased\",\n                                    config=my_config,)\n                                    .to(device)\n```\n\nBefore we can train our student model, we will need to **tokenize the dataset**. This is important because the models expect input text to be represented in a particular way.\nHere, I pad examples based on each batch's longest example. This allows the batches to be represented as a PyTorch tensor.\npython\n\n```\n# define text preprocessing\ndef preprocess_function(examples):\n    return tokenizer(examples[\"text\"], padding='max_length', truncation=True)\n\n# tokenize all datasetse\ntokenized_data = data.map(preprocess_function, batched=True)\ntokenized_data.set_format(type='torch', \n                          columns=['input_ids', 'attention_mask', 'labels'])\n```\n\nAnother important step before training is defining an evaluation strategy for our models during training. Below, I define a function that computes the **accuracy, precision, recall, and F1 score** given a model and dataset.\npython\n\n```\n# Function to evaluate model performance\ndef evaluate_model(model, dataloader, device):\n    model.eval()  # Set model to evaluation mode\n    all_preds = []\n    all_labels = []\n\n    # Disable gradient calculations\n    with torch.no_grad():\n        for batch in dataloader:\n            input_ids = batch['input_ids'].to(device)\n            attention_mask = batch['attention_mask'].to(device)\n            labels = batch['labels'].to(device)\n\n            # Forward pass to get logits\n            outputs = model(input_ids, attention_mask=attention_mask)\n            logits = outputs.logits\n\n            # Get predictions\n            preds = torch.argmax(logits, dim=1).cpu().numpy()\n            all_preds.extend(preds)\n            all_labels.extend(labels.cpu().numpy())\n\n    # Calculate evaluation metrics\n    accuracy = accuracy_score(all_labels, all_preds)\n    precision, recall, f1, _ = precision_recall_fscore_support(all_labels, \n                                                              all_preds, \n                                                              average='binary')\n\n    return accuracy, precision, recall, f1\n```\n\nNow, we are ready to begin the training process. To allow our student model to learn from both the ground truth labels in the training set (i.e., hard targets) and the teacher model's logits (i.e., soft targets), we must construct a special loss function that considers both targets.\nThis is done by **combining the KL divergence** of the student and teacher's output probability distribution **with the cross entropy loss** of the student's logits with the ground truth.\npython\n\n```\n# Function to compute distillation and hard-label loss\ndef distillation_loss(student_logits, teacher_logits, \n                      true_labels, temperature, alpha):\n    # Compute soft targets from teacher logits\n    soft_targets = nn.functional.softmax(teacher_logits / temperature, dim=1)\n    student_soft = nn.functional.log_softmax(student_logits / temperature, dim=1)\n\n    # KL Divergence loss for distillation\n    distill_loss = nn.functional.kl_div(student_soft, \n                                    soft_targets, \n                                    reduction='batchmean') * (temperature ** 2)\n\n    # Cross-entropy loss for hard labels\n    hard_loss = nn.CrossEntropyLoss()(student_logits, true_labels)\n\n    # Combine losses\n    loss = alpha * distill_loss + (1.0 - alpha) * hard_loss\n\n    return loss\n```\n\nNext, we define our hyperparameters, optimizer, and train/test datasets.\npython\n\n```\n# hyperparameters\nbatch_size = 32\nlr = 1e-4\nnum_epochs = 5\ntemperature = 2.0\nalpha = 0.5\n\n# define optimizer\noptimizer = optim.Adam(student_model.parameters(), lr=lr)\n\n# create training data loader\ndataloader = DataLoader(tokenized_data['train'], batch_size=batch_size)\n# create testing data loader\ntest_dataloader = DataLoader(tokenized_data['test'], batch_size=batch_size)\n```\n\nFinally, we train our student model using PyTorch.\npython\n\n```\n# put student model in train mode\nstudent_model.train()\n\n# train model\nfor epoch in range(num_epochs):\n    for batch in dataloader:\n        # Prepare inputs\n        input_ids = batch['input_ids'].to(device)\n        attention_mask = batch['attention_mask'].to(device)\n        labels = batch['labels'].to(device)\n\n        # Disable gradient calculation for teacher model\n        with torch.no_grad():\n            teacher_outputs = teacher_model(input_ids, \n                                            attention_mask=attention_mask)\n            teacher_logits = teacher_outputs.logits\n\n        # Forward pass through the student model\n        student_outputs = student_model(input_ids, \n                                        attention_mask=attention_mask)\n        student_logits = student_outputs.logits\n\n        # Compute the distillation loss\n        loss = distillation_loss(student_logits, teacher_logits, labels, \n                                  temperature, alpha)\n\n        # Backpropagation\n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n\n    print(f\"Epoch {epoch + 1} completed with loss: {loss.item()}\")\n\n    # Evaluate the teacher model\n    teacher_accuracy, teacher_precision, teacher_recall, teacher_f1 = \n                         evaluate_model(teacher_model, test_dataloader, device)\n\n    print(f\"Teacher (test) - Accuracy: {teacher_accuracy:.4f}, \n                              Precision: {teacher_precision:.4f}, \n                              Recall: {teacher_recall:.4f}, \n                              F1 Score: {teacher_f1:.4f}\")\n\n    # Evaluate the student model\n    student_accuracy, student_precision, student_recall, student_f1 = \n                         evaluate_model(student_model, test_dataloader, device)\n\n    print(f\"Student (test) - Accuracy: {student_accuracy:.4f}, \n                              Precision: {student_precision:.4f}, \n                              Recall: {student_recall:.4f}, \n                              F1 Score: {student_f1:.4f}\")\n    print(\"n\")\n\n    # put student model back into train mode\n    student_model.train()\n```\n\nThe training results are shown in the screenshot below. Remarkably, by the end of training, the **student model outperformed the teacher** across all evaluation metrics!\n![Knowledge distillation training results. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1-5iHN2Q3ZnrEBmucP1y8TQ.png)\nKnowledge distillation training results. Image by author.\nAs a final step, we can evaluate the models on the independent validation set, i.e., data not used in training model parameters or tuning hyperparameters.\npython\n\n```\n# create testing data loader\nvalidation_dataloader = DataLoader(tokenized_data['validation'], batch_size=8)\n\n# Evaluate the teacher model\nteacher_accuracy, teacher_precision, teacher_recall, teacher_f1 = \n                   evaluate_model(teacher_model, validation_dataloader, device)\nprint(f\"Teacher (validation) - Accuracy: {teacher_accuracy:.4f}, \n                               Precision: {teacher_precision:.4f}, \n                               Recall: {teacher_recall:.4f}, \n                               F1 Score: {teacher_f1:.4f}\")\n\n# Evaluate the student model\nstudent_accuracy, student_precision, student_recall, student_f1 = \n                   evaluate_model(student_model, validation_dataloader, device)\nprint(f\"Student (validation) - Accuracy: {student_accuracy:.4f}, \n                               Precision: {student_precision:.4f}, \n                               Recall: {student_recall:.4f}, \n                               F1 Score: {student_f1:.4f}\")\n```\n\nHere, again, we see the student outperform the teacher.\n![Model performances on the validation set. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1kajMTfpJTJYcFMu-8UWvfA.png)\nModel performances on the validation set. Image by author.\nSo far, we've **reduced our model from 109M parameters (438 MB) to 52.8M parameters (211 MB)**. However, we can go one step further and quantize the student model.\nFirst, we push the model of the [Hugging Face Hub](https://huggingface.co/shawhin/bert-phishing-classifier_student).\npython\n\n```\nstudent_model.push_to_hub(\"shawhin/bert-phishing-classifier_student\")\n```\n\nThen, we can load it back in using 4-bit quantization. For that, we can use the BitsAndBytes integration in the transformers library.\nWe set up the config to store model parameters using the **4-bit NormalFloat** **data type** described in the [QLoRA](https://medium.com/towards-data-science/qlora-how-to-fine-tune-an-llm-on-a-single-gpu-4e44d6b5be32) paper and the **bfloat16** **for computation** [10].\npython\n\n```\nfrom transformers import BitsAndBytesConfig\n\n# load model in model as 4-bit\nnf4_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype = torch.bfloat16,\n    bnb_4bit_use_double_quant=True\n)\n\nmodel_nf4 = AutoModelForSequenceClassification.from_pretrained(model_id, \n                                                device_map=device, \n                                                quantization_config=nf4_config)\n```\n\nWe can then evaluate our quantized model on the validation set.\npython\n\n```\n# Evaluate the student model\nquantized_accuracy, quantized_precision, quantized_recall, quantized_f1 = \n                       evaluate_model(model_nf4, validation_dataloader, device)\n\nprint(\"Post-quantization Performance\")\nprint(f\"Accuracy: {quantized_accuracy:.4f}, \n        Precision: {quantized_precision:.4f}, \n        Recall: {quantized_recall:.4f}, \n        F1 Score: {quantized_f1:.4f}\")\n```\n\n![Student model performance on the validation set after quantization. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1NtsbvduBzWt0V5puF2Eqfg.png)\nStudent model performance on the validation set after quantization. Image by author.\nOnce again, we see a small performance improvement after compression (down to 62.7MB). An intuitive explanation for this is **Occam's Razor principle** , which states that **simpler models are better**.\nIn this case, the model may be overparameterized for this binary classification task. Thus, simplifying the model results in better performance.\n## Recap\nWhile modern large language models (LLMs) demonstrate impressive performance on various tasks, their scale presents challenges in deploying them in real-world settings.\nRecent innovations in model compression techniques help mitigate these challenges by reducing the computational cost of LLM solutions. Here, we discussed three broad categories of compression techniques (Quantization, Pruning, and Knowledge Distillation) and walked through an example implementation in Python.\nMore on LLMs 👇\n> [**Large Language Models (LLMs)**](https://shawhin.medium.com/list/8e009ae3054c)\n**My website** : <https://www.shawhintalebi.com/>\n* * *\n[1] [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361)\n[2] [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://arxiv.org/abs/1710.09282)\n[3] [Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences](https://machinelearning.apple.com/research/model-compression-in-practice)\n[4] [Model Compression for Deep Neural Networks: A Survey](https://arxiv.org/abs/1710.09282)\n[5] [A Survey on Model Compression for Large Language Models](https://arxiv.org/abs/2308.07633)\n[6] [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://arxiv.org/abs/2402.17764)\n[7] [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://arxiv.org/abs/1710.01878)\n[8] [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531)\n[9] [Alpaca: A Strong, Replicable Instruction-Following Model](https://crfm.stanford.edu/2023/03/13/alpaca.html)\n[10] [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314)\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657074748_utev3i.jpeg)\nWritten By\n## [Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[See all from Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[AI](https://towardsdatascience.com/tag/ai/)[Hands On Tutorials](https://towardsdatascience.com/tag/hands-on-tutorials/)[Large Language Models](https://towardsdatascience.com/tag/large-language-models/)[Llm](https://towardsdatascience.com/tag/llm/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fcompressing-large-language-models-llms-9f406eea5b5e%2F&title=Compressing%20Large%20Language%20Models%20\\(LLMs\\))\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fcompressing-large-language-models-llms-9f406eea5b5e%2F&title=Compressing%20Large%20Language%20Models%20\\(LLMs\\))\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fcompressing-large-language-models-llms-9f406eea5b5e%2F&text=Compressing%20Large%20Language%20Models%20\\(LLMs\\))\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2018/08/1faRAVkapkq1IEYryH1RMPA.png)](https://towardsdatascience.com/math-behind-reinforcement-learning-the-easy-way-1b7ed0c030f4/)\n## [Math Behind Reinforcement Learning, the Easy Way](https://towardsdatascience.com/math-behind-reinforcement-learning-the-easy-way-1b7ed0c030f4/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nUpdate: The best way of learning and practicing Reinforcement Learning is by going to http://rl-lab.com...\n[Ziad SALLOUM](https://towardsdatascience.com/author/zsalloum/)\nAugust 2, 201810 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/18irf8y-s_TmtuoxyTIPWrA-scaled.jpeg)](https://towardsdatascience.com/aws-sagemaker-vs-azure-machine-learning-3ac0172495da/)\n## [AWS SageMaker vs Azure Machine Learning](https://towardsdatascience.com/aws-sagemaker-vs-azure-machine-learning-3ac0172495da/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nDuring my time as a consultant working in the analytics space I have had the opportunity to work on both AWS and Azure environments to...\n[Marco Susilo](https://towardsdatascience.com/author/marco-susilo/)\nDecember 10, 20203 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/19k1mU4sjdLbgm6UTk6lnQg.png)](https://towardsdatascience.com/how-to-make-bar-and-hbar-charts-with-labels-using-matplotlib-b701ce70ba9c/)\n## [How to make bar and hbar charts with labels using matplotlib](https://towardsdatascience.com/how-to-make-bar-and-hbar-charts-with-labels-using-matplotlib-b701ce70ba9c/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nBeginner data science guide using data from League of Legends\n[Jacob Toftgaard Rasmussen](https://towardsdatascience.com/author/jacobtoftgaardrasmussen/)\nJanuary 8, 202110 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/01/0H4B7uAjhFKmY1wjW-scaled.jpg)](https://towardsdatascience.com/interpretable-machine-learning-models-aef0c7be3fd9/)\n## [Interpretable Machine Learning Models](https://towardsdatascience.com/interpretable-machine-learning-models-aef0c7be3fd9/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nA thorough introduction to the interpretation of logistic regression models and decision trees.\n[Hennie de Harder](https://towardsdatascience.com/author/hennie-de-harder/)\nJanuary 4, 20206 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/1DooRfkQAWQ1RHSE7PjNqIg-scaled.jpeg)](https://towardsdatascience.com/5-key-differences-between-data-scientists-and-data-engineers-4fb67bb5e4b6/)\n## [5 Key Differences Between Data Scientists and Data Engineers](https://towardsdatascience.com/5-key-differences-between-data-scientists-and-data-engineers-4fb67bb5e4b6/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nDistinguish the two building blocks of data science\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nDecember 31, 20205 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/04/img1222.jpg)](https://towardsdatascience.com/kv-cache-is-eating-your-vram-heres-how-google-fixed-it-with-turboquant/)\n## [KV Cache Is Eating Your VRAM. Here’s How Google Fixed It With TurboQuant.](https://towardsdatascience.com/kv-cache-is-eating-your-vram-heres-how-google-fixed-it-with-turboquant/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nA visual and technical guide to the TurboQuant workflow: Solving the VRAM bottleneck with randomized rotation and residual correction\n[Aman Vasisht](https://towardsdatascience.com/author/aman.vasisht/)\nApril 19, 202610 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/07bdlgzmt-ISTNA12.jpg)](https://towardsdatascience.com/matrix-and-vector-operations-in-logistic-regression-e35714c4810f/)\n## [Matrix and Vector Operations in Logistic Regression](https://towardsdatascience.com/matrix-and-vector-operations-in-logistic-regression-e35714c4810f/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nVectorized Logistic Regression\n[Murali Kashaboina](https://towardsdatascience.com/author/murali-kashaboina/)\nJuly 7, 20239 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/01/0esTC8qaUqP7xeRbJ.jpg)](https://towardsdatascience.com/machine-learning-project-predicting-boston-house-prices-with-regression-b4e47493633d/)\n## [Machine Learning Project: Predicting Boston House Prices With Regression](https://towardsdatascience.com/machine-learning-project-predicting-boston-house-prices-with-regression-b4e47493633d/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nLearn how to apply regression to solve a real-world problem\n[Victor Roman](https://towardsdatascience.com/author/rromanss23/)\nJanuary 20, 201917 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/07/09SJ-pW25cKm1pXD.jpg)](https://towardsdatascience.com/the-deep-connection-between-boosting-and-lasso-b40c00fc71c6/)\n## [The Deep Connection Between Boosting and LASSO](https://towardsdatascience.com/the-deep-connection-between-boosting-and-lasso-b40c00fc71c6/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nUnderstanding a powerful algorithm through a radical perspective shift\n[Sankar Srinivasan](https://towardsdatascience.com/author/s1sriniv/)\nJuly 7, 202218 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n",
  "metadata": {
    "title": "Compressing Large Language Models (LLMs) | Towards Data Science",
    "description": "Make LLMs 10X smaller without sacrificing performance",
    "keywords": null,
    "author": "Shaw Talebi",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Compressing Large Language Models (LLMs)",
    "og:description": "Make LLMs 10X smaller without sacrificing performance",
    "og:url": "https://towardsdatascience.com/compressing-large-language-models-llms-9f406eea5b5e/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1VDHOHFG-PA8VddO-tuTFbA.png",
    "og:image:width": "1280",
    "og:image:height": "720",
    "og:image:type": "image/png",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Shaw Talebi",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "11 min read",
    "article:published_time": "2024-08-30T04:09:49.000Z",
    "article:modified_time": "2024-08-30T04:09:49.000Z",
    "article:author": "https://towardsdatascience.com/author/shawhin/"
  },
  "paywall": {
    "listed_paid": true,
    "detected": false,
    "marker": "",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 202,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_000.txt",
      "chars": 370
    },
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_001.txt",
      "chars": 60
    },
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_002.txt",
      "chars": 335
    },
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_003.txt",
      "chars": 347
    },
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_004.txt",
      "chars": 346
    },
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_005.txt",
      "chars": 1130
    },
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_006.txt",
      "chars": 810
    },
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_007.txt",
      "chars": 375
    },
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_008.txt",
      "chars": 2107
    },
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_009.txt",
      "chars": 974
    },
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_010.txt",
      "chars": 70
    },
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_011.txt",
      "chars": 468
    },
    {
      "language": "txt",
      "file": "attachments/1164e5b8f6f3/code/block_012.txt",
      "chars": 396
    }
  ],
  "images": [
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1VDHOHFG-PA8VddO-tuTFbA.png",
      "alt": "Image from Canva.",
      "local_path": "attachments/1164e5b8f6f3/images/000.png",
      "bytes": 720955,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1M_0ZN9vERSCayZID56g1UA.png",
      "alt": "Quantization analogy. Image by author.",
      "local_path": "attachments/1164e5b8f6f3/images/001.png",
      "bytes": 277813,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1l9mhfjOIRnohnycafFXLIw.png",
      "alt": "Pruning analogy. Image by author.",
      "local_path": "attachments/1164e5b8f6f3/images/002.png",
      "bytes": 191027,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1-5iHN2Q3ZnrEBmucP1y8TQ.png",
      "alt": "Knowledge distillation training results. Image by author.",
      "local_path": "attachments/1164e5b8f6f3/images/003.png",
      "bytes": 37671,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1kajMTfpJTJYcFMu-8UWvfA.png",
      "alt": "Model performances on the validation set. Image by author.",
      "local_path": "attachments/1164e5b8f6f3/images/004.png",
      "bytes": 7286,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/1NtsbvduBzWt0V5puF2Eqfg.png",
      "alt": "Student model performance on the validation set after quantization. Image by author.",
      "local_path": "attachments/1164e5b8f6f3/images/005.png",
      "bytes": 6211,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2018/08/1faRAVkapkq1IEYryH1RMPA.png",
      "alt": "",
      "local_path": "attachments/1164e5b8f6f3/images/006.png",
      "bytes": 61810,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/18irf8y-s_TmtuoxyTIPWrA-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/1164e5b8f6f3/images/007.jpeg",
      "bytes": 488746,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/19k1mU4sjdLbgm6UTk6lnQg.png",
      "alt": "",
      "local_path": "attachments/1164e5b8f6f3/images/008.png",
      "bytes": 4066,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/01/0H4B7uAjhFKmY1wjW-scaled.jpg",
      "alt": "",
      "local_path": "attachments/1164e5b8f6f3/images/009.jpg",
      "bytes": 164187,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/1DooRfkQAWQ1RHSE7PjNqIg-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/1164e5b8f6f3/images/010.jpeg",
      "bytes": 1057498,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/04/img1222.jpg",
      "alt": "",
      "local_path": "attachments/1164e5b8f6f3/images/011.jpg",
      "bytes": 135595,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/07bdlgzmt-ISTNA12.jpg",
      "alt": "",
      "local_path": "attachments/1164e5b8f6f3/images/012.jpg",
      "bytes": 131110,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/01/0esTC8qaUqP7xeRbJ.jpg",
      "alt": "",
      "local_path": "attachments/1164e5b8f6f3/images/013.jpg",
      "bytes": 124647,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/07/09SJ-pW25cKm1pXD.jpg",
      "alt": "",
      "local_path": "attachments/1164e5b8f6f3/images/014.jpg",
      "bytes": 72403,
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "src": "https://www.youtube.com/c/TowardsDataScience",
      "quality": "unavailable",
      "local_path": "",
      "note": "无1080p且无720p，未交付更低清晰度"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/large-language-models/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/shawhin/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/qlora-how-to-fine-tune-an-llm-on-a-single-gpu-4e44d6b5be32"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/ai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/hands-on-tutorials/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/large-language-models/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/llm/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/math-behind-reinforcement-learning-the-easy-way-1b7ed0c030f4/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/zsalloum/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/aws-sagemaker-vs-azure-machine-learning-3ac0172495da/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/marco-susilo/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://shawhin.medium.com/list/large-language-models-llms-8e009ae3054c"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-engineer"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/machine-learning"
    },
    {
      "type": "external",
      "href": "https://github.com/ShawhinT/YouTube-Blog/tree/main/LLMs/model-compression"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/shawhin/bert-phishing-classifier_teacher"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/shawhin/bert-phishing-classifier_student"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/shawhin/bert-phishing-classifier_student_4bit"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/datasets/shawhin/phishing-site-classification"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/google-bert/bert-base-uncased"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/distilbert/distilbert-base-uncased"
    },
    {
      "type": "external",
      "href": "https://medium.com/towards-data-science/qlora-how-to-fine-tune-an-llm-on-a-single-gpu-4e44d6b5be32"
    },
    {
      "type": "external",
      "href": "https://shawhin.medium.com/list/8e009ae3054c"
    },
    {
      "type": "external",
      "href": "https://www.shawhintalebi.com/"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2001.08361"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/1710.09282"
    },
    {
      "type": "external",
      "href": "https://machinelearning.apple.com/research/model-compression-in-practice"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2308.07633"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T11:11:49Z",
  "source_url": "https://towardsdatascience.com/compressing-large-language-models-llms-9f406eea5b5e/",
  "delivery_status": "excluded_paid",
  "discovery_status": "excluded_paid",
  "missing_reason": "按原约定排除付费/会员全文"
}