{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Philipp Schmid",
  "title_listed": "GPTQ Llama",
  "keywords": "GPTQ,量化,Llama",
  "domain": "大模型算法",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://www.philschmid.de/gptq-llama",
  "publish_date": "2023-08",
  "corpus_stage": "推理部署",
  "notes": "GPTQ量化Llama模型",
  "id": "e5b1fe377b43",
  "collected_at": "2026-10-02T14:29:13",
  "title": "Optimize open LLMs using GPTQ and Hugging Face Optimum",
  "body_markdown": "Optimize open LLMs using GPTQ and Hugging Face Optimum\nAugust 31, 2023\n9\nminute read\nView Code\nThe Hugging Face Optimum team collaborated with AutoGPTQ library to provide a simple API that apply GPTQ quantization on language models. With GPTQ quantization open LLMs to 8, 4, 3 or even 2 bits to run them on smaller Hardware without a big drop of performance.\nIn the blog, you will learn how to:\nSetup our development environment\nPrepare quantization dataset\nLoad and Quantize Model\nTest performance and inference speed\nBonus: Run Inference with Text Generation Inference\nBut we before we get started lets take quick look on what GPTQ does.\nNote: This tutorial was created and run on a g5.2xlarge AWS EC2 Instance, including an NVIDIA A10G GPU.\nWhat is GPTQ?\nGPTQ\nis a post-training quantziation method to compress LLMs, like GPT. GPTQ compresses GPT models by reducing the number of bits needed to store each weight in the model, from 32 bits down to just 3-4 bits. This means the model takes up much less memory, so it can run on less Hardware, e.g. Single GPU for 13B Llama2 models. GPTQ analyzes each layer of the model separately and approximating the weights in a way that preserves the overall accuracy.\nThe main benefits are:\nQuantizes the weights of the model layer-by-layer to 4 bits instead of 16 bits, this reduces the needed memory by 4x.\nQuantization is done gradually to minimize the accuracy loss from quantization.\nAchieves same latency as fp16 model, but 4x less memory usage, sometimes faster due to custom kernels, e.g.\nExllama\nQuantized weights can be saved to disk for a head of time quantization.\nNote: GPTQ quantization only works for text model for now. Futhermore, the quantization process can take a lot of time. You check on the\nHugging Face Hub\nif there is not already a GPTQ quantized version of the model you want to use.\n1. Setup our development environment\nLet's start coding, but first, install our dependencies.\nPython\n!\npip install\n\"transformers[sentencepiece]==4.32.1\"\n\"optimum==1.12.0\"\n\"auto-gptq==0.4.2\"\n\"accelerate==0.22.0\"\n\"safetensors>=0.3.1\"\n--\nupgrade\n2. Prepare quantization dataset\nGPTQ is a post-training quantization method, so we need to prepare a dataset to quantize our model. We can either use a dataset from the\nHugging Face Hub\nor use our own dataset. In this blog, we are going to use the\nWikiText\ndataset from the Hugging Face Hub. The dataset is used to quantize the weights to minimize the performance loss. It is recommended to use a quantization dataset with atleast\n128\nsamples.\nNote:\nTheBloke\na very active community member is contributing hundreds of gptq weights to the Hugging Face Hub. He mostly uses wikitext as quantization dataset for general domain models.\nIf you want to use, e.g. your fine-tuning dataset for quantization you can provide it as a list instead of the \"id\", check out this\nexample\n.\nPython\n# Dataset id from Hugging Face\ndataset_id =\n\"wikitext2\"\n3. Load and Quantize Model\nOptimum integrates GPTQ quantization in the\noptimum.qptq\nnamespace with a\nGPTQQuantizer\n. The quantizer takes our dataset (id or list), bits, and model_seqlen as input. For more customization check\nhere\n.\nPython\nfrom\noptimum.gptq\nimport\nGPTQQuantizer\n# GPTQ quantizer\nquantizer =\nGPTQQuantizer\n(\nbits\n=\n4\n,\ndataset\n=dataset_id,\nmodel_seqlen\n=\n4096\n)\nquantizer.quant_method =\n\"gptq\"\nAfter we have created our Quantizer we can load our model using Transformers. In our example we will quantize a\nLlama 2 7B\n, which we trained in my other blog post\n\"Extended Guide: Instruction-tune Llama 2\"\n. We are going to load our model in\nfp16\nsince GPTQ adopts a mixed int4/fp16 quantization scheme where weights are quantized as int4 while activations remain in float16.\nPython\nimport\ntorch\nfrom\ntransformers\nimport\nAutoModelForCausalLM, AutoTokenizer\n# Hugging Face model id\nmodel_id =\n\"philschmid/llama-2-7b-instruction-generator\"\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(model_id,\nuse_fast\n=\nFalse\n)\n# bug with fast tokenizer\nmodel = AutoModelForCausalLM.\nfrom_pretrained\n(model_id,\nlow_cpu_mem_usage\n=\nTrue\n,\ntorch_dtype\n=torch.float16)\n# we load the model in fp16 on purpose\nAfter we loaded our model we are ready to quantize it.\nNote: Quantization can take process can take a lot of time depending on one's hardware. For this example the quantization on a single A10G GPU for a 7B model took ~45minutes.\nPython\nimport\nos\nimport\njson\n# quantize the model\nquantized_model = quantizer.\nquantize_model\n(model, tokenizer)\n# save the quantize model to disk\nsave_folder =\n\"quantized_llama\"\nquantized_model.\nsave_pretrained\n(save_folder,\nsafe_serialization\n=\nTrue\n)\n# load fresh, fast tokenizer and save it to disk\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(model_id).\nsave_pretrained\n(save_folder)\n# save quantize_config.json for TGI\nwith\nopen(os.path.\njoin\n(save_folder,\n\"quantize_config.json\"\n),\n\"w\"\n,\nencoding\n=\n\"utf-8\"\n)\nas\nf:\nquantizer.disable_exllama =\nFalse\njson.\ndump\n(quantizer.\nto_dict\n(), f,\nindent\n=\n2\n)\nsince the model was partially offloaded it set\ndisable_exllama\nto\nTrue\nto avoid an error. For inference and production load we want to leverage the exllama kernels. Therefore we need to change the\nconfig.json\nPython\nwith\nopen(os.path.\njoin\n(save_folder,\n\"config.json\"\n),\n\"r\"\n,\nencoding\n=\n\"utf-8\"\n)\nas\nf:\nconfig = json.\nload\n(f)\nconfig[\n\"quantization_config\"\n][\n\"disable_exllama\"\n] =\nFalse\nwith\nopen(os.path.\njoin\n(save_folder,\n\"config.json\"\n),\n\"w\"\n,\nencoding\n=\n\"utf-8\"\n)\nas\nf:\njson.\ndump\n(config, f,\nindent\n=\n2\n)\n4. Test performance and inference speed\nSince the latest release of transformers we can load any GPTQ quantized model directly using the\nAutoModelForCausalLM\nclass this. You can either load already quantized models from Hugging Face, e.g.\nTheBloke/Llama-2-13B-chat-GPTQ\nor models you quantized yourself. Since we want to test here the results of our quantization we are going to load our quantized model from disk and compare it to our non quantize model.\nFirst lets our our non quantized model and test it on a simple prompt.\nPython\nimport\ntime\n# The prompt is based on the fine-tuning from the model: https://www.philschmid.de/instruction-tune-llama-2#4-test-model-and-run-inference\nprompt =\n\"\"\"### Instruction:\nUse the Input below to create an instruction, which could have been used to generate the input using an LLM.\n### Input:\nDear [boss name],\nI'm writing to request next week, August 1st through August 4th,\noff as paid time off.\nI have some personal matters to attend to that week that require\nme to be out of the office. I wanted to give you as much advance\nnotice as possible so you can plan accordingly while I am away.\nThank you, [Your name]\n### Response:\n\"\"\"\n# helper function to generate text and measure latency\ndef\ngenerate_helper\n(\npipeline\n,\nprompt\n=prompt):\n# warm up\nfor\ni\nin\nrange(\n5\n):\n_ =\npipeline\n(\n\"Warm up\"\n)\n# measure latency in a simple way\nstart = time.\ntime\n()\nout =\npipeline\n(prompt,\nmax_new_tokens\n=\n100\n,\ndo_sample\n=\nTrue\n,\ntop_p\n=\n0.9\n,\ntemperature\n=\n0.9\n)\nend = time.\ntime\n()\ngenerated_text = out[\n0\n][\n\"generated_text\"\n][len(prompt):]\nlatency_per_token_in_ms = ((end-start)/len(pipeline.\ntokenizer\n(generated_text)[\n\"input_ids\"\n]))*\n1000\n# return the generated text and the latency\nreturn\n{\n\"text\"\n: out[\n0\n][\n\"generated_text\"\n][len(prompt):],\n\"latency\"\n:\nf\n\"\n{\nround(latency_per_token_in_ms,\n2\n)\n}\nms/token\"\n}\nWe can load the vanilla transformers model and run inference using the\npipeline\nclass.\nPython\nimport\ntorch\nfrom\ntransformers\nimport\nAutoModelForCausalLM, AutoTokenizer, pipeline\n# Hugging Face model id\nmodel_id =\n\"philschmid/llama-2-7b-instruction-generator\"\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(model_id)\nmodel = AutoModelForCausalLM.\nfrom_pretrained\n(model_id,\ndevice_map\n=\n\"auto\"\n,\ntorch_dtype\n=torch.float16)\n# we load the model in fp16 on purpose\npipe =\npipeline\n(\n\"text-generation\"\n,\nmodel\n=model,\ntokenizer\n=tokenizer)\nlets create our vanilla base line\nPython\nimport\ntorch\nvanilla_res =\ngenerate_helper\n(pipe)\nprint(\nf\n\"Latency:\n{\nvanilla_res[\n'latency'\n]\n}\n\"\n)\nprint(\nf\n\"GPU memory:\n{\ntorch.cuda.\nmemory_allocated\n() /\n1024\n**\n3\n:.2f\n}\nGB\"\n)\nprint(\nf\n\"Generated Instruction:\n{\nvanilla_res[\n'text'\n]\n}\n\"\n)\n# Latency: 37.49ms/token\n# GPU memory: 12.62 GB\n# Generated Instruction: Write a request for PTO letter to my boss\nPython\n# clean up\ndel\npipe\ndel\nmodel\ndel\ntokenizer\ntorch.cuda.\nempty_cache\n()\nSince we have now our baseline we can test and validate our GPTQ quantize weights. Therefore we will use the new\ngptq\nintegration into the\nAutoModelForCausalLM\nclass where we can directly load the\ngptq\nweights.\nPython\nimport\ntorch\nfrom\ntransformers\nimport\nAutoModelForCausalLM, AutoTokenizer, pipeline\n# path to gptq weights\nmodel_id =\n\"quantized_llama\"\nq_tokenizer = AutoTokenizer.\nfrom_pretrained\n(model_id)\nq_model = AutoModelForCausalLM.\nfrom_pretrained\n(model_id,\ndevice_map\n=\n\"auto\"\n,\ntorch_dtype\n=torch.float16)\nqtq_pipe =\npipeline\n(\n\"text-generation\"\n,\nmodel\n=q_model,\ntokenizer\n=q_tokenizer)\nNow, we can test our quantized model on the same prompt as our baseline.\nPython\ngpq_res =\ngenerate_helper\n(qtq_pipe)\nprint(\nf\n\"Latency:\n{\ngpq_res[\n'latency'\n]\n}\n\"\n)\nprint(\nf\n\"GPU memory:\n{\ntorch.cuda.\nmemory_allocated\n() /\n1024\n**\n3\n:.2f\n}\nGB\"\n)\nprint(\nf\n\"Generated Instruction:\n{\ngpq_res[\n'text'\n]\n}\n\"\n)\n# Latency: 36.0ms/token\n# GPU memory: 3.83 GB\n# Generated Instruction: Write a letter requesting time off\nFor comparison the vanilla model needed ~12.6GB Memory and the GPTQ model needed ~3.8GB Memory, with equal performance. GPTQ allowed us to save ~4x memory (don't forget pytorch has default kernels).\n5. Bonus: Run Inference with Text Generation Inference\nText Generation Inference supports GPTQ model for more efficient deployments. We simply need to provide\ngptq\nas\nQUANTIZE\nenvironment variable when starting our container.\nPython\nmodel=\n\"/home/ubuntu/test-gptq\"\nnum_shard=\n1\nquantize=\n\"gptq\"\nmax_input_length=\n1562\nmax_total_tokens=\n4096\n# 4096\n!\ndocker run\n--\ngpus all -ti -p\n8080\n:\n80\n\\\n-e\nMODEL_ID\n=\n$\nmodel \\\n-e\nQUANTIZE\n=\n$\nquantize \\\n-e\nNUM_SHARD\n=\n$\nnum_shard \\\n-e\nMAX_INPUT_LENGTH\n=\n$\nmax_input_length \\\n-e\nMAX_TOTAL_TOKENS\n=\n$\nmax_total_tokens \\\n-v\n$\nmodel:\n$\nmodel \\\nghcr.io/huggingface/text-generation-inference:\n1.0\n.3\nWe can invoke our container using curl.\n_Note: The first request will be slow. _\nPython\ncurl\n127.0\n.0.1:\n8080\n/generate \\\n-X\nPOST\n\\\n-d\n'{\"inputs\":\"### Instruction:\n\\n\nUse the Input below to create an instruction, which could have been used to generate the input using an LLM.\n\\n\\n\n### Input:\n\\n\nDear [boss name],\n\\n\\n\nI am writing to request next week, August 1st through August 4th,\n\\n\noff as paid time off.\n\\n\\n\nI have some personal matters to attend to that week that require\n\\n\nme to be out of the office. I wanted to give you as much advance\n\\n\nnotice as possible so you can plan accordingly while I am away.\n\\n\\n\nThank you, [Your name]\n\\n\\n\n### Response:\",\"parameters\":{\"temperature\":0.2, \"top_p\": 0.95, \"max_new_tokens\": 256}}'\n\\\n-H\n'Content-Type: application/json'\nWith Text Generation inference we are achieving ~\n22.942983ms\nlatency per token, which is 2x faster than transformers. If you plan to deploy your model in production, I would recommend to use Text Generation Inference.\nThanks for reading! If you have any questions, feel free to contact me on\nTwitter\nor\nLinkedIn\n.",
  "metadata": {
    "title": "Optimize open LLMs using GPTQ and Hugging Face Optimum",
    "description": "Learn how to quantize Llama 2 7B with GPTQ to use 4x less memory.",
    "keywords": null,
    "author": "Philipp Schmid",
    "og:title": "Optimize open LLMs using GPTQ and Hugging Face Optimum",
    "og:description": "Learn how to quantize Llama 2 7B with GPTQ to use 4x less memory.",
    "og:url": "https://www.philschmid.de/gptq-llama",
    "og:image": "https://www.philschmid.de/static/blog/gptq-llama/thumbnail.jpg",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "Optimize open LLMs using GPTQ and Hugging Face Optimum",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Optimize open LLMs using GPTQ and Hugging Face Optimum",
    "twitter:description": "Learn how to quantize Llama 2 7B with GPTQ to use 4x less memory.",
    "twitter:image": "https://www.philschmid.de/static/blog/gptq-llama/thumbnail.jpg",
    "article:published_time": "2023-08-31"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/e5b1fe377b43/code/body-000.txt",
      "chars": 139,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-001.txt",
      "chars": 55,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-002.txt",
      "chars": 162,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-003.txt",
      "chars": 397,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-004.txt",
      "chars": 585,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-005.txt",
      "chars": 281,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-006.txt",
      "chars": 1332,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-007.txt",
      "chars": 431,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-008.txt",
      "chars": 339,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-009.txt",
      "chars": 68,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-010.txt",
      "chars": 375,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-011.txt",
      "chars": 307,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-012.txt",
      "chars": 387,
      "language": "text"
    },
    {
      "file": "attachments/e5b1fe377b43/code/body-013.txt",
      "chars": 673,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/philschmid/deep-learning-pytorch-huggingface/blob/main/training/optimize-llama-2-gptq.ipynb",
      "label": "View Code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2210.17323",
      "label": "GPTQ",
      "type": "a"
    },
    {
      "href": "https://github.com/turboderp/exllama",
      "label": "Exllama",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/models?search=gptq",
      "label": "Hugging Face Hub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets",
      "label": "Hugging Face Hub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/wikitext",
      "label": "WikiText",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/TheBloke",
      "label": "TheBloke",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/drive/1_TIrmuKOFhuRRiTWN94iLKUFu6ZX4ceb",
      "label": "example",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/optimum/blob/234a427450a7dcc978b227fa627ebcdab1764318/optimum/gptq/quantizer.py",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/philschmid/llama-2-7b-instruction-generator",
      "label": "Llama 2 7B",
      "type": "a"
    },
    {
      "href": "https://www.philschmid.de/instruction-tune-llama-2",
      "label": "\"Extended Guide: Instruction-tune Llama 2\"",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/TheBloke/Llama-2-13B-chat-GPTQ",
      "label": "TheBloke/Llama-2-13B-chat-GPTQ",
      "type": "a"
    },
    {
      "href": "https://twitter.com/_philschmid",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/philipp-schmid-a6a2bb196/",
      "label": "LinkedIn",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:48Z",
  "source_url": "https://www.philschmid.de/gptq-llama",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2210.17323",
      "bytes": 521818,
      "reused": true,
      "local_path": "papers/repair/af396e9b1e330cf9d61c.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 11272,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/e5b1fe377b43/pages/source.html",
  "local_html": "attachments/e5b1fe377b43/pages/article.html",
  "body_base_path": "attachments/e5b1fe377b43/pages/",
  "body_markdown_path": "attachments/e5b1fe377b43/pages/article.md"
}