{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "Hugging Face Blog",
  "title_listed": "Making LLMs even more accessible with bitsandbytes 4-bit quantization",
  "keywords": "量化,4-bit,LLM",
  "domain": "大模型算法",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://huggingface.co/blog/4bit-transformers-bitsandbytes",
  "publish_date": "2023-07",
  "corpus_stage": "推理部署",
  "notes": "4-bit量化让LLM更易访问",
  "id": "ceaaacdb3460",
  "collected_at": "2026-10-02T15:23:46",
  "title": "Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA",
  "body_markdown": "Back to Articles\nMaking LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA\nPublished\n\t\t\t\t\tMay 24, 2023\nUpdate on GitHub\nUpvote\n182\n+176\nYounes B\nybelkada\nFollow\nTim Dettmers\ntimdettmers\nFollow\nguest\nArtidoro Pagnoni\nartidoro\nFollow\nguest\nSylvain Gugger\nsgugger\nFollow\nSourab Mangrulkar\nsmangrul\nFollow\nLLMs are known to be large, and running or training them in consumer hardware is a huge challenge for users and accessibility. \nOur\nLLM.int8 blogpost\nshowed how the techniques in the\nLLM.int8 paper\nwere integrated in transformers using the\nbitsandbytes\nlibrary.\nAs we strive to make models even more accessible to anyone, we decided to collaborate with bitsandbytes again to allow users to run models in 4-bit precision. This includes a large majority of HF models, in any modality (text, vision, multi-modal, etc.). Users can also train adapters on top of 4bit models leveraging tools from the Hugging Face ecosystem. This is a new method introduced today in the QLoRA paper by Dettmers et al. The abstract of the paper is as follows:\nWe present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights (b) double quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) paged optimizers to manage memory spikes. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across 8 instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to ChatGPT. We release all of our models and code, including CUDA kernels for 4-bit training.\nResources\nThis blogpost and release come with several resources to get started with 4bit models and QLoRA:\nOriginal paper\nBasic usage Google Colab notebook\n- This notebook shows how to use 4bit models in inference with all their variants, and how to run GPT-neo-X (a 20B parameter model) on a free Google Colab instance 🤯\nFine tuning Google Colab notebook\n- This notebook shows how to fine-tune a 4bit model on a downstream task using the Hugging Face ecosystem. We show that it is possible to fine tune GPT-neo-X 20B on a Google Colab instance!\nOriginal repository for replicating the paper's results\nGuanaco 33b playground\n- or check the playground section below\nIntroduction\nIf you are not familiar with model precisions and the most common data types (float16, float32, bfloat16, int8), we advise you to carefully read the introduction in\nour first blogpost\nthat goes over the details of these concepts in simple terms with visualizations.\nFor more information we recommend reading the fundamentals of floating point representation through\nthis wikibook document\n.\nThe recent QLoRA paper explores different data types, 4-bit Float and 4-bit NormalFloat. We will discuss here the 4-bit Float data type since it is easier to understand.\nFP8 and FP4 stand for Floating Point 8-bit and 4-bit precision, respectively. They are part of the minifloats family of floating point values (among other precisions, the minifloats family also includes bfloat16 and float16).\nLet’s first have a look at how to represent floating point values in FP8 format, then understand how the FP4 format looks like.\nFP8 format\nAs discussed in  our previous blogpost, a floating point contains n-bits, with each bit falling into a specific category that is responsible for representing a component of the number (sign, mantissa and exponent). These represent the following.\nThe FP8 (floating point 8) format has been first introduced in the paper\n“FP8 for Deep Learning”\nwith two different FP8 encodings: E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa).\nOverview of Floating Point 8 (FP8) format.  Source: Original content from\nsgugger\nAlthough the precision is substantially reduced by reducing the number of bits from 32 to 8, both versions can be used in a variety of situations. Currently one could use\nTransformer Engine library\nthat is also integrated with HF ecosystem through accelerate.\nThe potential floating points that can be represented in the E4M3 format are in the range -448 to 448, whereas in the E5M2 format, as the number of bits of the exponent increases, the range increases to -57344 to 57344 - but with a loss of precision because the number of possible representations remains constant.\nIt has been empirically proven that the E4M3 is best suited for the forward pass, and the second version is best suited for the backward computation\nFP4 precision in a few words\nThe sign bit represents the sign (+/-), the exponent bits a base two to the power of the integer represented by the bits (e.g.\n2^{010} = 2^{2} = 4\n), and the fraction or mantissa is the sum of powers of negative two which are “active” for each bit that is “1”. If a bit is “0” the fraction remains unchanged for that power of\n2^-i\nwhere i is the position of the bit in the bit-sequence. For example, for mantissa bits 1010 we have\n(0 + 2^-1 + 0 + 2^-3) = (0.5 + 0.125) = 0.625\n. To get a value, we add\n1\nto the fraction and multiply all results together, for example, with 2 exponent bits and one mantissa bit the representations 1101 would be:\n-1 * 2^(2) * (1 + 2^-1) = -1 * 4 * 1.5 = -6\nFor FP4 there is no fixed format and as such one can try combinations of different mantissa/exponent combinations. In general, 3 exponent bits do a bit better in most cases. But sometimes 2 exponent bits and a mantissa bit yield better performance.\nQLoRA paper, a new way of democratizing quantized large transformer models\nIn few words, QLoRA reduces the memory usage of LLM finetuning without performance tradeoffs compared to standard 16-bit model finetuning. This method enables 33B model finetuning on a single 24GB GPU and 65B model finetuning on a single 46GB GPU.\nMore specifically, QLoRA uses 4-bit quantization to compress a pretrained language model. The LM parameters are then frozen and a relatively small number of trainable parameters are added to the model in the form of Low-Rank Adapters. During finetuning, QLoRA backpropagates gradients through the frozen 4-bit quantized pretrained language model into the Low-Rank Adapters. The LoRA layers are the only parameters being updated during training. Read more about LoRA in the\noriginal LoRA paper\n.\nQLoRA has one storage data type (usually 4-bit NormalFloat) for the base model weights and a computation data type (16-bit BrainFloat) used to perform computations. QLoRA dequantizes weights from the storage data type to the computation data type to perform the forward and backward passes, but only computes weight gradients for the LoRA parameters which use 16-bit bfloat. The weights are decompressed only when they are needed, therefore the memory usage stays low during training and inference.\nQLoRA tuning is shown to match 16-bit finetuning methods in a wide range of experiments. In addition, the Guanaco models, which use QLoRA finetuning for LLaMA models on the\nOpenAssistant dataset (OASST1)\n, are state-of-the-art chatbot systems and are close to ChatGPT on the Vicuna benchmark. This is an additional demonstration of the power of QLoRA tuning.\nFor a more detailed reading, we recommend you read the\nQLoRA paper\n.\nHow to use it in transformers?\nIn this section let us introduce the transformers integration of this method, how to use it and which models can be effectively quantized.\nGetting started\nAs a quickstart, load a model in 4bit by (at the time of this writing) installing accelerate and transformers from source, and make sure you have installed the latest version of bitsandbytes library (0.39.0).\npip install -q -U bitsandbytes\npip install -q -U git+https://github.com/huggingface/transformers.git\npip install -q -U git+https://github.com/huggingface/peft.git\npip install -q -U git+https://github.com/huggingface/accelerate.git\nQuickstart\nThe basic way to load a model in 4bit is to pass the argument\nload_in_4bit=True\nwhen calling the\nfrom_pretrained\nmethod by providing a device map (pass\n\"auto\"\nto get a device map that will be automatically inferred).\nfrom\ntransformers\nimport\nAutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\n\"facebook/opt-350m\"\n, load_in_4bit=\nTrue\n, device_map=\n\"auto\"\n)\n...\nThat's all you need!\nAs a general rule, we recommend users to not manually set a device once the model has been loaded with\ndevice_map\n. So any device assignment call to the model, or to any model’s submodules should be avoided after that line - unless you know what you are doing.\nKeep in mind that loading a quantized model will automatically cast other model's submodules into\nfloat16\ndtype. You can change this behavior, (if for example you want to have the layer norms in\nfloat32\n), by passing\ntorch_dtype=dtype\nto the\nfrom_pretrained\nmethod.\nAdvanced usage\nYou can play with different variants of 4bit quantization such as NF4 (normalized float 4 (default)) or pure FP4 quantization. Based on theoretical considerations and empirical results from the paper, we recommend using NF4 quantization for better performance.\nOther options include\nbnb_4bit_use_double_quant\nwhich uses a second quantization after the first one to save an additional 0.4 bits per parameter. And finally, the compute type. While 4-bit bitsandbytes stores weights in 4-bits, the computation still happens in 16 or 32-bit and here any combination can be chosen (float16, bfloat16, float32 etc).\nThe matrix multiplication and training will be faster if one uses a 16-bit compute dtype (default torch.float32). One should leverage the recent\nBitsAndBytesConfig\nfrom transformers to change these parameters. An example to load a model in 4bit using NF4 quantization below with double quantization with the compute dtype bfloat16 for faster training:\nfrom\ntransformers\nimport\nBitsAndBytesConfig\n\n\nnf4_config = BitsAndBytesConfig(\n   load_in_4bit=\nTrue\n,\n   bnb_4bit_quant_type=\n\"nf4\"\n,\n   bnb_4bit_use_double_quant=\nTrue\n,\n   bnb_4bit_compute_dtype=torch.bfloat16\n)\n\nmodel_nf4 = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=nf4_config)\nChanging the compute dtype\nAs mentioned above, you can also change the compute dtype of the quantized model by just changing the\nbnb_4bit_compute_dtype\nargument in\nBitsAndBytesConfig\n.\nimport\ntorch\nfrom\ntransformers\nimport\nBitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(\n   load_in_4bit=\nTrue\n,\n   bnb_4bit_compute_dtype=torch.bfloat16\n)\nNested quantization\nFor enabling nested quantization, you can use the\nbnb_4bit_use_double_quant\nargument in\nBitsAndBytesConfig\n. This will enable a second quantization after the first one to save an additional 0.4 bits per parameter. We also use this feature in the training Google colab notebook.\nfrom\ntransformers\nimport\nBitsAndBytesConfig\n\ndouble_quant_config = BitsAndBytesConfig(\n   load_in_4bit=\nTrue\n,\n   bnb_4bit_use_double_quant=\nTrue\n,\n)\n\nmodel_double_quant = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=double_quant_config)\nAnd of course, as mentioned in the beginning of the section, all of these components are composable. You can combine all these parameters together to find the optimal use case for you. A rule of thumb is: use double quant if you have problems with memory, use NF4 for higher precision, and use a 16-bit dtype for faster finetuning. For instance in the\ninference demo\n, we use nested quantization, bfloat16 compute dtype and NF4 quantization to fit gpt-neo-x-20b (40GB) entirely in 4bit in a single 16GB GPU.\nCommon questions\nIn this section, we will also address some common questions anyone could have regarding this integration.\nDoes FP4 quantization have any hardware requirements?\nNote that this method is only compatible with GPUs, hence it is not possible to quantize models in 4bit on a CPU. Among GPUs, there should not be any hardware requirement about this method, therefore any GPU could be used to run the 4bit quantization as long as you have CUDA>=11.2 installed. \nKeep also in mind that the computation is not done in 4bit, the weights and activations are compressed to that format and the computation is still kept in the desired or native dtype.\nWhat are the supported models?\nSimilarly as the integration of LLM.int8 presented in\nthis blogpost\nthe integration heavily relies on the\naccelerate\nlibrary. Therefore, any model that supports accelerate loading (i.e. the\ndevice_map\nargument when calling\nfrom_pretrained\n) should be quantizable in 4bit. Note also that this is totally agnostic to modalities, as long as the models can be loaded with the\ndevice_map\nargument, it is possible to quantize them.\nFor text models, at this time of writing, this would include most used architectures such as Llama, OPT, GPT-Neo, GPT-NeoX for text models, Blip2 for multimodal models, and so on.\nAt this time of writing, the models that support accelerate are:\n[\n'bigbird_pegasus'\n,\n'blip_2'\n,\n'bloom'\n,\n'bridgetower'\n,\n'codegen'\n,\n'deit'\n,\n'esm'\n,\n'gpt2'\n,\n'gpt_bigcode'\n,\n'gpt_neo'\n,\n'gpt_neox'\n,\n'gpt_neox_japanese'\n,\n'gptj'\n,\n'gptsan_japanese'\n,\n'lilt'\n,\n'llama'\n,\n'longformer'\n,\n'longt5'\n,\n'luke'\n,\n'm2m_100'\n,\n'mbart'\n,\n'mega'\n,\n'mt5'\n,\n'nllb_moe'\n,\n'open_llama'\n,\n'opt'\n,\n'owlvit'\n,\n'plbart'\n,\n'roberta'\n,\n'roberta_prelayernorm'\n,\n'rwkv'\n,\n'switch_transformers'\n,\n't5'\n,\n'vilt'\n,\n'vit'\n,\n'vit_hybrid'\n,\n'whisper'\n,\n'xglm'\n,\n'xlm_roberta'\n]\nNote that if your favorite model is not there, you can open a Pull Request or raise an issue in transformers to add the support of accelerate loading for that architecture.\nCan we train 4bit/8bit models?\nIt is not possible to perform pure 4bit training on these models. However, you can train these models by leveraging parameter efficient fine tuning methods (PEFT) and train for example adapters on top of them. That is what is done in the paper and is officially supported by the PEFT library from Hugging Face. We also provide a\ntraining notebook\nand recommend users to check the\nQLoRA repository\nif they are interested in replicating the results from the paper.\nThe output activations original (frozen) pretrained weights (left) are augmented by a low rank adapter comprised of weight matrics A and B (right).\nWhat other consequences are there?\nThis integration can open up several positive consequences to the community and AI research as it can affect multiple use cases and possible applications. \nIn RLHF (Reinforcement Learning with Human Feedback) it is possible to load a single base model, in 4bit and train multiple adapters on top of it, one for the reward modeling, and another for the value policy training. A more detailed blogpost and announcement will be made soon about this use case.\nWe have also made some benchmarks on the impact of this quantization method on training large models on  consumer hardware. We have run several experiments on finetuning 2 different architectures, Llama 7B (15GB in fp16) and Llama 13B (27GB in fp16) on an NVIDIA T4 (16GB) and here are the results\nModel name\nHalf precision model size (in GB)\nHardware type / total VRAM\nquantization method (CD=compute dtype / GC=gradient checkpointing / NQ=nested quantization)\nbatch_size\ngradient accumulation steps\noptimizer\nseq_len\nResult\n<10B scale models\ndecapoda-research/llama-7b-hf\n14GB\n1xNVIDIA-T4 / 16GB\nLLM.int8 (8-bit) + GC\n1\n4\nAdamW\n512\nNo OOM\ndecapoda-research/llama-7b-hf\n14GB\n1xNVIDIA-T4 / 16GB\nLLM.int8 (8-bit) + GC\n1\n4\nAdamW\n1024\nOOM\ndecapoda-research/llama-7b-hf\n14GB\n1xNVIDIA-T4 / 16GB\n4bit + NF4 + bf16 CD + no GC\n1\n4\nAdamW\n512\nNo OOM\ndecapoda-research/llama-7b-hf\n14GB\n1xNVIDIA-T4 / 16GB\n4bit + FP4 + bf16 CD + no GC\n1\n4\nAdamW\n512\nNo OOM\ndecapoda-research/llama-7b-hf\n14GB\n1xNVIDIA-T4 / 16GB\n4bit + NF4 + bf16 CD + no GC\n1\n4\nAdamW\n1024\nOOM\ndecapoda-research/llama-7b-hf\n14GB\n1xNVIDIA-T4 / 16GB\n4bit + FP4 + bf16 CD + no GC\n1\n4\nAdamW\n1024\nOOM\ndecapoda-research/llama-7b-hf\n14GB\n1xNVIDIA-T4 / 16GB\n4bit + NF4 + bf16 CD + GC\n1\n4\nAdamW\n1024\nNo OOM\n10B+ scale models\ndecapoda-research/llama-13b-hf\n27GB\n2xNVIDIA-T4 / 32GB\nLLM.int8 (8-bit) + GC\n1\n4\nAdamW\n512\nNo OOM\ndecapoda-research/llama-13b-hf\n27GB\n1xNVIDIA-T4 / 16GB\nLLM.int8 (8-bit) + GC\n1\n4\nAdamW\n512\nOOM\ndecapoda-research/llama-13b-hf\n27GB\n1xNVIDIA-T4 / 16GB\n4bit + FP4 + bf16 CD + no GC\n1\n4\nAdamW\n512\nOOM\ndecapoda-research/llama-13b-hf\n27GB\n1xNVIDIA-T4 / 16GB\n4bit + FP4 + fp16 CD + no GC\n1\n4\nAdamW\n512\nOOM\ndecapoda-research/llama-13b-hf\n27GB\n1xNVIDIA-T4 / 16GB\n4bit + NF4 + fp16 CD + GC\n1\n4\nAdamW\n512\nNo OOM\ndecapoda-research/llama-13b-hf\n27GB\n1xNVIDIA-T4 / 16GB\n4bit + NF4 + fp16 CD + GC\n1\n4\nAdamW\n1024\nOOM\ndecapoda-research/llama-13b-hf\n27GB\n1xNVIDIA-T4 / 16GB\n4bit + NF4 + fp16 CD + GC + NQ\n1\n4\nAdamW\n1024\nNo OOM\nWe have used the recent\nSFTTrainer\nfrom TRL library, and the benchmarking script can be found\nhere\nPlayground\nTry out the Guananco model cited on the paper on\nthe playground\nor directly below\nAcknowledgements\nThe HF team would like to acknowledge all the people involved in this project from University of Washington, and for making this available to the community.\nThe authors would also like to thank\nPedro Cuenca\nfor kindly reviewing the blogpost,\nOlivier Dehaene\nand\nOmar Sanseviero\nfor their quick and strong support for the integration of the paper's artifacts on the HF Hub.\nDatasets mentioned in this article\n2\nSpaces mentioned in this article\n1\nMore Articles from our Blog\nguide\nquantization\ntransformers\nQuanto: a PyTorch quantization backend for Optimum\n45\nMarch 18, 2024\nllm\noptimization\nquantization\nOverview of natively supported quantization schemes in 🤗 Transformers\n+1\n13\nSeptember 12, 2023\nCommunity\nAMonrroy\nMar 7, 2025\nhola\nReply\nliam255\nDec 17, 2025\n•\nedited Feb 23\nReally cool insights! Makes me want to push the limits of my own chatbots. With CustomGPT.ai, I’ve been able to build bots that understand my content deeply and respond naturally. Seeing tools like QLoRA and 4-bit models makes advanced LLMs feel way more approachable for anyone experimenting\nSee translation\nReply\nEdit\nPreview\nUpload images, audio, and videos by dragging in the text input, pasting, or\nclicking here\n.\nTap or paste here to upload images\nComment\n·\nSign up\nor\nlog in\nto comment\nUpvote\n182\n+170\nDatasets mentioned in this article\n2\nSpaces mentioned in this article\n1",
  "metadata": {
    "title": "Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA",
    "description": "We’re on a journey to advance and democratize artificial intelligence through open source and open science.",
    "keywords": null,
    "author": null,
    "og:title": "Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA",
    "og:description": "We’re on a journey to advance and democratize artificial intelligence through open source and open science.",
    "og:type": "website",
    "og:url": "https://huggingface.co/blog/4bit-transformers-bitsandbytes",
    "og:image": "https://huggingface.co/blog/assets/96_hf_bitsandbytes_integration/Thumbnail_blue.png",
    "twitter:card": "summary_large_image",
    "twitter:site": "@huggingface",
    "twitter:image": "https://huggingface.co/blog/assets/96_hf_bitsandbytes_integration/Thumbnail_blue.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/ceaaacdb3460/code/body-000.txt",
      "chars": 231,
      "language": "text"
    },
    {
      "file": "attachments/ceaaacdb3460/code/body-001.txt",
      "chars": 155,
      "language": "text"
    },
    {
      "file": "attachments/ceaaacdb3460/code/body-002.txt",
      "chars": 301,
      "language": "text"
    },
    {
      "file": "attachments/ceaaacdb3460/code/body-003.txt",
      "chars": 165,
      "language": "text"
    },
    {
      "file": "attachments/ceaaacdb3460/code/body-004.txt",
      "chars": 256,
      "language": "text"
    },
    {
      "file": "attachments/ceaaacdb3460/code/body-005.txt",
      "chars": 474,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/635f7cae9ba251bf848f6f5d/eLBmnVTYRdTvVe_lT88vR.jpeg",
      "bytes": 9908,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-59b5f1dd99bd94402ac5.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/635f7cae9ba251bf848f6f5d/eLBmnVTYRdTvVe_lT88vR.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/600988773adca878cda1d294/-fm8VEoWgqFvwfYbpRhEF.jpeg",
      "bytes": 4416,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-3dfc328374b9f87fa114.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/600988773adca878cda1d294/-fm8VEoWgqFvwfYbpRhEF.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/fb8d31d179d62dc95e4f4374713d0245.svg",
      "bytes": 1281,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-b73de5b13be31c5db4f9.svg",
      "src": "https://huggingface.co/avatars/fb8d31d179d62dc95e4f4374713d0245.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/8a9d1e9811f840ffc96f734fafe52be3.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-f76c7b5b1a8bc863e97d.svg",
      "src": "https://huggingface.co/avatars/8a9d1e9811f840ffc96f734fafe52be3.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/f94b783c8f49019ebb0a69f6e3053b77.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-b0caa1abe63989e58cb2.svg",
      "src": "https://huggingface.co/avatars/f94b783c8f49019ebb0a69f6e3053b77.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/af693217938fa021e318df1f79ae6d2d.svg",
      "bytes": 1284,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-21a989bf61cfa79f63ee.svg",
      "src": "https://huggingface.co/avatars/af693217938fa021e318df1f79ae6d2d.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/bitsandbytes/FP8-scheme.png",
      "bytes": 470057,
      "reused": true,
      "local_path": "attachments/ceaaacdb3460/images/005.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/bitsandbytes/FP8-scheme.png",
      "alt": "fp8_scheme",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/blog/133_trl_peft/lora-animated.gif",
      "bytes": 322447,
      "reused": true,
      "local_path": "attachments/ceaaacdb3460/images/006.gif",
      "src": "https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/blog/133_trl_peft/lora-animated.gif",
      "alt": "lora-gif",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/blog/assets/169_quanto_intro/thumbnail.png",
      "bytes": 884047,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-136961a0a72fdef08123.png",
      "src": "https://huggingface.co/blog/assets/169_quanto_intro/thumbnail.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/647995564be04c76ce4547b3/-tlmMQgr-CTm5TCIjGApB.jpeg",
      "bytes": 9960,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-f5ff89c1827e16893114.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/647995564be04c76ce4547b3/-tlmMQgr-CTm5TCIjGApB.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png",
      "bytes": 6460,
      "reused": true,
      "local_path": "attachments/ceaaacdb3460/images/000.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/63ce875d199b36f7552d4f07/bpUrvhXDagzRqZ3vxTcSF.jpeg",
      "bytes": 13840,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-832904037d4458dfc5dd.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/63ce875d199b36f7552d4f07/bpUrvhXDagzRqZ3vxTcSF.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/blog/assets/163_overview_quantization_transformers/thumbnail.jpg",
      "bytes": 265214,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-fba6432d26f723d0528e.jpg",
      "src": "https://huggingface.co/blog/assets/163_overview_quantization_transformers/thumbnail.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1642598610696-noauth.jpeg",
      "bytes": 14704,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-5d683e6b0de2cae1c12e.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1642598610696-noauth.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1644340617257-noauth.png",
      "bytes": 15504,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-caf460c770ad66b65f7f.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1644340617257-noauth.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/9849aa31fa73a2d59dde2d2a6b9af4b7.svg",
      "bytes": 1282,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-c9a7cb64032ff911ee6f.svg",
      "src": "https://huggingface.co/avatars/9849aa31fa73a2d59dde2d2a6b9af4b7.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VFO_-tSfESObMKW9jPv5_.png",
      "bytes": 6210,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-a58b8243d2c21e5bfdfa.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VFO_-tSfESObMKW9jPv5_.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/b174053b24740d662f9d8c2dbac36faf.svg",
      "bytes": 1283,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-d6fe996fc4437bd1e540.svg",
      "src": "https://huggingface.co/avatars/b174053b24740d662f9d8c2dbac36faf.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/65fb32f58a66679f92b73458/ozQW6W27tVCM-SRouzGJa.jpeg",
      "bytes": 7118,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-c760922a486eb036e802.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/65fb32f58a66679f92b73458/ozQW6W27tVCM-SRouzGJa.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/63581bce0e4fef2198254575/yMzMAZCXYul2fRKHpbcvx.png",
      "bytes": 6290,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-c9d0e7cdbbeb67877fa0.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/63581bce0e4fef2198254575/yMzMAZCXYul2fRKHpbcvx.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/a454cde25975abd09acb20067b447720.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/ceaaacdb3460/images/body-0141bb53cee8661cbd01.svg",
      "src": "https://huggingface.co/avatars/a454cde25975abd09acb20067b447720.svg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://huggingface.co/blog",
      "label": "Back to Articles",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/blog/blob/main/4bit-transformers-bitsandbytes.md",
      "label": "Update on GitHub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/login?next=%2Fblog%2F4bit-transformers-bitsandbytes",
      "label": "Upvote 182",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/helincesxyz",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/nezubn",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/xianghang",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/lengxingxin",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/jeongah",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/cctristan",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/ybelkada",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/timdettmers",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/artidoro",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/sgugger",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/smangrul",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/hf-bitsandbytes-integration",
      "label": "LLM.int8 blogpost",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2208.07339",
      "label": "LLM.int8 paper",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.14314",
      "label": "Original paper",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/drive/1ge2F1QSK8Q7h0hn3YKuBCOAS0bK8E0wf?usp=sharing",
      "label": "Basic usage Google Colab notebook",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/drive/1VoYNfYDKcKRQRor98Zbf2-9VQTtGJ24k?usp=sharing",
      "label": "Fine tuning Google Colab notebook",
      "type": "a"
    },
    {
      "href": "https://github.com/artidoro/qlora",
      "label": "Original repository for replicating the paper's results",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi",
      "label": "Guanaco 33b playground",
      "type": "a"
    },
    {
      "href": "https://en.wikibooks.org/wiki/A-level_Computing/AQA/Paper_2/Fundamentals_of_data_representation/Floating_point_numbers",
      "label": "this wikibook document",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2209.05433.pdf",
      "label": "“FP8 for Deep Learning”",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/bitsandbytes/FP8-scheme.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/NVIDIA/TransformerEngine",
      "label": "Transformer Engine library",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2106.09685",
      "label": "original LoRA paper",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/OpenAssistant/oasst1",
      "label": "OpenAssistant dataset (OASST1)",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/blog/133_trl_peft/lora-animated.gif",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://gist.github.com/younesbelkada/f48af54c74ba6a39a7ae4fd777e72fe8",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/pcuenq",
      "label": "Pedro Cuenca",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/olivierdehaene",
      "label": "Olivier Dehaene",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/osanseviero",
      "label": "Omar Sanseviero",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/quanto-introduction",
      "label": "guide quantization transformers Quanto: a PyTorch quantization backend for Optimum 45 March 18, 2024",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/overview-quantization-transformers",
      "label": "llm optimization quantization Overview of natively supported quantization schemes in 🤗 Transformers +1 13 September 12, ",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/AMonrroy",
      "label": "AMonrroy",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/liam255",
      "label": "liam255",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/join?next=%2Fblog%2F4bit-transformers-bitsandbytes",
      "label": "Sign up",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/wendysWenda",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/ManniX-ITA",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/marcsun13",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/jucamohedano",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/fordacre",
      "label": "",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:23Z",
  "source_url": "https://huggingface.co/blog/4bit-transformers-bitsandbytes",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2208.07339",
      "bytes": 719021,
      "reused": true,
      "local_path": "papers/repair/30fa6cd91bee75957c34.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.14314",
      "bytes": 1065470,
      "reused": true,
      "local_path": "papers/repair/df2f754581ddf63a5d62.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2209.05433.pdf",
      "bytes": 281195,
      "reused": false,
      "local_path": "papers/repair/da36c2f46941d6653450.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2106.09685",
      "bytes": 1609513,
      "reused": true,
      "local_path": "papers/repair/e7d2fed957472d7677ec.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 19460,
  "expected_body_images": 21,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ceaaacdb3460/pages/source.html",
  "local_html": "attachments/ceaaacdb3460/pages/article.html",
  "body_base_path": "attachments/ceaaacdb3460/pages/",
  "body_markdown_path": "attachments/ceaaacdb3460/pages/article.md"
}