{
"source_file": "0930",
"source_platform": "Medium",
"author": "Naser Tamimi",
"title_listed": "Fine Tuning LLMs on a Single Consumer Graphic Card",
"keywords": "",
"domain": "大模型算法",
"content_type": "深度技术解析",
"is_paid_flag": "否",
"url": "https://towardsdatascience.com/fine-tuning-llms-on-a-single-consumer-graphic-card-6de1587daddb/",
"publish_date": "2024-01-31",
"corpus_stage": "",
"notes": "",
"id": "7b6a4f796cca",
"collected_at": "2026-10-02T05:01:55",
"title": "Fine Tuning LLMs on a Single Consumer Graphic Card | Towards Data Science",
"body_markdown": "\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences \nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# Fine Tuning LLMs on a Single Consumer Graphic Card\nLearnings from fine-tuning a large language model on a single consumer GPU\n[Naser Tamimi](https://towardsdatascience.com/author/tamimi-naser/)\nJanuary 31, 202410 min read\n### GENERATIVE AI\n\nImage by Author (Midjourney).\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) and [Computer Science](https://roadmap.sh/computer-science) roadmaps.\n## Background\nWhen we think about Large Language Models or any other generative models, the first hardware that comes to mind is GPU. Without GPUs, many advancements in Generative AI, machine learning, deep learning, and data science would've been impossible. If 15 years ago, gamers were enthusiastic about the latest GPU technologies, today data scientists and machine learning engineers join them and pursue the news in this field too. Although usually gamers and ML users are looking at two different kinds of GPUs and graphic cards.\nGaming users usually use consumer graphic cards (such as NVIDIA GeForce RTX Series GPUs), while ML and AI developers usually follow news about Data Center and Cloud Computing GPUs (such as V100, A100, or H100). Gaming graphic cards usually have much less GPU memory (at most 24GB as of January 2024) compared to Data Center GPUs (in the range of 40GB to 80GB usually). Also, their price is another significant difference. While most consumer graphic cards could be up to $3000, most Data Center graphic cards start from that price and can go tens of thousands of dollars easily.\nSince many people, including myself, might have a consumer graphic card for their gaming or daily use, they might be interested to see if they can use the same graphic cards for training, fine-tuning, or inference of LLM models. In 2020, I wrote a comprehensive article about whether we can use consumer graphic cards for data science projects ([link to the article](https://medium.com/datadriveninvestor/an-affordable-gpu-for-data-scientists-db376450117)). At that time, the models were mostly small ML or Deep Learning models and even a graphic card with 6GB of memory could handle many training projects. But, in this article, I am going to use such a graphic card for large language models with billions of parameters.\nFor this article, I used my GeForce 3090 RTX card which has 24GB of GPU memory. For your reference, data center graphic cards such as A100 and H100 have 40GB and 80GB of memory respectively. Also, a typical AWS EC2 p4d.24xlarge instance has 8 GPUs (V100) with a total of 320GB of GPU memory. As you can see the difference between a simple consumer GPU and a typical cloud ml instance is significant. But the question is, can we train large models on single consumer graphic cards or not? And if yes, what are the tips and learnings? Read the rest of this article, to find out.\n## Hardware and Software Setup\nBefore loading any LLM model or training dataset, we need to find out what hardware and software we need for such a process.\nAs mentioned, I used the NVIDIA GeForce RTX 3090 GPU because it has one of the highest memory (24GB) among consumer GPUs (FYI: 4090 model has the same memory size too). It is based on Ampere Architecture which is the same architecture that famous A100 GPUs have. You can see more about GeForce RTX 3090 GPU specifications [here](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/).\nAfter all the tests, I believe 24GB is the least amount of GPU memory that we need for working with LLMs with billions of parameters.\n\nImage by Author\nIn addition to the graphics card, we need to make sure that our PC has a good ventilation system. During fine-tuning, the temperature of the GPU easily goes up, and its fans are not enough to keep it cool. Higher GPU temperature can lower the GPU performance and the process will take much longer time.\nIn addition to hardware, t[here](https://github.com/Keith-Hon/bitsandbytes-windows) are some software considerations that I must mention here. First of all, if you are a Windows user, I have a piece of bad news for you. Some libraries and tools only work on Linux. Specifically, bitsandbytes which is being used frequently for model quantization is not Windows-friendly. Some people made a wrapper for Windows (for example here), but they have their cons and pros. So, my advice is either to install Linux on WSL or like me have a dual boot system and fully switch to Linux while you are working on LLMs.\nAlso, you need to install PyTorch and a compatible CUDA version. My recommendation is to install CUDA 12.3 ([link](https://developer.nvidia.com/cuda-downloads)). Then you need to go to this page, and based on your system, CUDA version, and package manager system, download and install the correct PyTorch ().\n> Note: If you are using CUDA 12.3, you might need to add or configure BNB_CUDA_VERSION and LD_LIBRARY_PATH env variables in your .bashrc file. Here is an example for your reference.\nbash\n\n```\nexport BNB_CUDA_VERSION=123\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home//local/cuda-12.3\n```\n\nAnd finally, you need to install the following packages on your system. I recommend creating a new virtual environment (venv) to avoid conflict with other installed packages on your system. Also, for your reference here are versions of packages that I successfully used:\npython\n\n```\ntorch==2.1.2\ntransformers==4.36.2\ndatasets==2.16.1\nbitsandbytes==0.42.0\npeft==0.7.1\n```\n\n## Technical Background\nNow that you have all the hardware and software ready for working with LLMs on your system, it is better to have a very brief review of the technical concepts that you will face in the next sections.\nLarge language models comprise millions or billions of parameters. Usually, we use pre-trained models that have been trained on billions and sometimes trillions of tokens after a long training process and usually millions of dollars in cost. Each of these model parameters takes 32 bits (4 bytes) of memory for loading. As a rule of thumb, each 1 billion parameters require about 4GB of memory to just load the model. One technique of using less memory for loading (and later inferencing or training a model later) is \"Quantization\". In this technique, we reduce the precision of the model weights from 32-bit full precision (fb32) to 16-bit (fp16 or bfloat16), 8-bit (int8), or even less ([read more](https://huggingface.co/blog/4bit-transformers-bitsandbytes)). As you can imagine by reducing the precision of the model weights we can load larger models into a limited memory, but this comes at the cost of reducing the model's performance. However, some studies suggested that the model performance difference between fp32 and bfloat16 is insignificant and many famous models (including Llama2) were pre-trained on bfloat16.\nQuantization is a technique that we must use when we are fine-tuning or inferencing a large language model on a single GPU with 24GB of memory. Later you will see that the `bitsandbytes` library can help us achieve model quantization.\nEven with the most rigorous quantization techniques, we are still unable to pre-train a small-size LLM model with millions of parameters. Chris Fregley et al. have a good rule of thumb in their newly published [Generative AI on AWS](https://www.amazon.com/Generative-AWS-Context-Aware-Multimodal-Applications/dp/1098159225/ref=sr_1_1?crid=OSEFPN00F1WQ&keywords=generative+ai+on+aws&qid=1705896435&sprefix=generativ%2Caps%2C153&sr=8-1) book for required memory for training a model. As they explained for each 1 billion parameters of a model, we need 6GB of memory (in 16-bit half precision) to load and train the model. Remember, the memory size is only a part of the training story. The amount of time needed to complete pre-training is another important part. As an example, the smallest Llama2 model (which is Llama2 7B) has 7 billion parameters and it took 184320 GPU hours to complete its training ([read more](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md)).\nTherefore, most people (even with significant hardware resources and budget) prefer to use a pre-trained model and only fine-tune it for their specific use case. Still, the process of full fine-tuning could be overwhelming with limited resources (such as a single GPU). Therefore, Paremer-Efficient Fine-Tuning (PEFT), which only updates a limited subset of model parameters, looks more realistic with less computed resources.\nAmong different PEFT techniques, LoRA (Low Ranking Adaption) is very popular because of its computing efficiency. In this technique, we freeze all the original model weights and instead, we train low-rank matrices that can be added to specific layers of the Transformer architecture ([read more](https://arxiv.org/abs/2106.09685)). In many cases, using LoRA for fine-tuning an LLM, we update 0.5% of model weights.\nQLoRA is a variation of LoRA that combines LoRA with the Quantization concept that we explained. Specifically, in our QLoRA implementation, we will use nf4 or Normal Float 4 for fine-tuning the model. QLoRA is very helpful in our case study of fine-tuning a large model with a single consumer GPU.\n## Coding Time\nFinally, it's coding time!\nYou can find the working Jupyter Notebook [here on my GitHub repo](https://github.com/tamiminaser/llm-single-gpu/blob/main/qlora_mistral_7b.ipynb). For many parts of this code, I got inspired and followed the instructions by Mathieu Busquet [in this neat article](https://blog.ovhcloud.com/fine-tuning-llama-2-models-using-a-single-gpu-qlora-and-ai-notebooks/).\nI will not discuss the code line by line, but I will emphasize parts of the code that are important for fine-tuning a large model on a single GPU.\n### Transformer Model\nFirst of all, I chose the [Mistral 7B model](https://huggingface.co/mistralai/Mistral-7B-v0.1) (`mistralai/Mistral-7B-v0.1`) for this testing. The Mistral 7B model developed by Mistral AI is an open-source LLM released in September 2023 ([link to the paper](https://arxiv.org/abs/2310.06825)). From many aspects, this model outperforms famous models such as Llama2 (see the following charts).\n\nImage from Mistral Release Blog Post ()\n### Dataset\nAlso, I used Databricks databricks-dolly-15k dataset (under CC BY-SA 3.0 license) for fine-tuning ([read more](https://huggingface.co/datasets/databricks/databricks-dolly-15k)). I used a small subset (1000 rows) of this data to reduce the fine-tuning time and proof of concept.\n### Configurations\nAt the model loading time, I used the following quantization config to overcome the GPU memory limitations that I am facing.\npython\n\n```\nquantization_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type=\"nf4\",\n bnb_4bit_compute_dtype=torch.bfloat16,\n)\n```\n\nThis quantization config is vital for fine-tuning a model on a single GPUT as it has one low-precision storage data type, `nf4` (4-bit Normal Float) which is a 4-bit data type, and a computation data type which is `bfloat16`. In practice, this means whenever a QLORA weight tensor is used, we dequantize the tensor to `bfloat16` and then perform matrix multiplication in 16-bit (read more in [the original paper](https://arxiv.org/abs/2305.14314)).\nAlso, as mentioned before, we are using LoRA in conjunction with Quantization, known as QLoRA, to overcome the memory limitations. Here are my configs for the LoRA:\npython\n\n```\nlora_config = LoraConfig(\n r=16,\n lora_alpha=64,\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\"], \n bias=\"none\",\n lora_dropout=0.05,\n task_type=\"CAUSAL_LM\",\n)\n```\n\nFor my LoRA configuration, I used 16 for rank. It is advised to set the rank between 4 and 16 to get a good trade-off between reducing the number of trainable parameters and the model performance. Finally, we applied LoRA to a subset of linear layers in our Mistral 7B transformer model.\n## Training and Monitoring\nUsing my single graphic card, I could complete 4 epochs of training (1000 steps). For me, one of the purposes of doing such a test, training an LLM on a single local GPU, is to monitor the hardware resources without any restrictions. One of the simplest tools for monitoring GPU during training is the Nvidia System Management Interface (SMI). Simply open a terminal and in your command line type:\nbash\n\n```\nnvidia-smi\n```\n\nor for continuously monitoring and updating, use (refreshes every 1 sec):\nbash\n\n```\nnvidia-smi -l 1\n```\n\nAs a result, you will see the memory usage by each process on your GPU. In the following SMI view, I just loaded the model and it took about 5GB of memory (thanks to the quantization). Also, as you see the model is loaded by the Anaconda3 Python (a Jupyter notebook) process.\ntext\n\n```\n+---------------------------------------------------------------------------------------+\n| NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 |\n|-----------------------------------------+----------------------+----------------------+\n| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |\n| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |\n| | | MIG M. |\n|=========================================+======================+======================|\n| 0 NVIDIA GeForce RTX 3090 On | 00000000:29:00.0 On | N/A |\n| 30% 37C P8 33W / 350W | 5346MiB / 24576MiB | 5% Default |\n| | | N/A |\n+-----------------------------------------+----------------------+----------------------+\n\n+---------------------------------------------------------------------------------------+\n| Processes: |\n| GPU GI CI PID Type Process name GPU Memory |\n| ID ID Usage |\n|=======================================================================================|\n| 0 N/A N/A 1610 G /usr/lib/xorg/Xorg 179MiB |\n| 0 N/A N/A 1820 G /usr/bin/gnome-shell 41MiB |\n| 0 N/A N/A 108004 G ...2023.3.3/host-linux-x64/nsys-ui.bin 8MiB |\n| 0 N/A N/A 168032 G ...seed-version=20240110-180219.406000 117MiB |\n| 0 N/A N/A 327503 C /home/***/anaconda3/bin/python 4880MiB |\n+---------------------------------------------------------------------------------------+\n```\n\nAnd here (the following snapshot) is the memory state after about 30 steps into the training process. As you see, the used GPU memory is now about 15GB.\ntext\n\n```\n+---------------------------------------------------------------------------------------+\n| NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 |\n|-----------------------------------------+----------------------+----------------------+\n| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |\n| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |\n| | | MIG M. |\n|=========================================+======================+======================|\n| 0 NVIDIA GeForce RTX 3090 On | 00000000:29:00.0 On | N/A |\n| 30% 57C P2 341W / 350W | 15054MiB / 24576MiB | 100% Default |\n| | | N/A |\n+-----------------------------------------+----------------------+----------------------+\n\n+---------------------------------------------------------------------------------------+\n| Processes: |\n| GPU GI CI PID Type Process name GPU Memory |\n| ID ID Usage |\n|=======================================================================================|\n| 0 N/A N/A 1610 G /usr/lib/xorg/Xorg 179MiB |\n| 0 N/A N/A 1820 G /usr/bin/gnome-shell 40MiB |\n| 0 N/A N/A 108004 G ...2023.3.3/host-linux-x64/nsys-ui.bin 8MiB |\n| 0 N/A N/A 168032 G ...seed-version=20240110-180219.406000 182MiB |\n| 0 N/A N/A 327503 C /home/***/anaconda3/bin/python 14524MiB |\n+---------------------------------------------------------------------------------------+\n```\n\nAlthough SMI is a simple tool for monitoring GPU memory usage, there are still a few advanced monitoring tools that provide more detailed information. One of these advanced tools is PyTorch Memory Snapshot which you can read more about in [this interesting article](https://pytorch.org/blog/understanding-gpu-memory-1/?utm_content=275432244&utm_medium=social&utm_source=linkedin&hss_channel=lcp-78618366).\n## Summary\nIn this article, I showed you that it is possible to fine-tune a large language model such as Mistral 7B on a single 24 GB GPU (such as NVIDIA GeForce RTX 3090 GPU). However, as discussed in detail, special PEFT techniques like QLoRA are necessary. Also, the batch size of the model is important and we might need much longer training time, just because of our limited resources.\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n\nWritten By\n## [Naser Tamimi](https://towardsdatascience.com/author/tamimi-naser/)\n[See all from Naser Tamimi](https://towardsdatascience.com/author/tamimi-naser/)\n[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Gpu](https://towardsdatascience.com/tag/gpu/)[Llm](https://towardsdatascience.com/tag/llm/)[Programming](https://towardsdatascience.com/tag/programming/)[Python](https://towardsdatascience.com/tag/python/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-llms-on-a-single-consumer-graphic-card-6de1587daddb%2F&title=Fine%20Tuning%20LLMs%20on%20a%20Single%20Consumer%20Graphic%20Card)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-llms-on-a-single-consumer-graphic-card-6de1587daddb%2F&title=Fine%20Tuning%20LLMs%20on%20a%20Single%20Consumer%20Graphic%20Card)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-llms-on-a-single-consumer-graphic-card-6de1587daddb%2F&text=Fine%20Tuning%20LLMs%20on%20a%20Single%20Consumer%20Graphic%20Card)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [](https://towardsdatascience.com/geospatial-analysis-in-python-and-jupyter-notebooks-f90de25b0777/)\n## [GeoSpatial analysis in Python and Jupyter Notebooks](https://towardsdatascience.com/geospatial-analysis-in-python-and-jupyter-notebooks-f90de25b0777/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nGeospatial analysis of Barcelona's bike rental service (bicing), using geopandas and kepler.gl.\n[Fran Pérez](https://towardsdatascience.com/author/franperez/)\nOctober 20, 20206 min read\n * [](https://towardsdatascience.com/intuition-and-implementation-of-non-max-suppression-algorithm-in-object-detection-d68ba938b630/)\n## [Intuition and Implementation of Non-Max Suppression Algorithm in Object Detection](https://towardsdatascience.com/intuition-and-implementation-of-non-max-suppression-algorithm-in-object-detection-d68ba938b630/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nThis article explains the usage and implementation of the Non-Max Suppression technique used for object detection in an image\n[Prateek Chhikara](https://towardsdatascience.com/author/prateekchhikara/)\nFebruary 1, 20225 min read\n * [](https://towardsdatascience.com/what-i-learned-coding-google-from-scratch-cd7ddb02d9/)\n## [What I Learned Coding Google From Scratch](https://towardsdatascience.com/what-i-learned-coding-google-from-scratch-cd7ddb02d9/)\n[Programming](https://towardsdatascience.com/category/programming/)\nHow to accelerate your programming development\n[Kevin Babitz](https://towardsdatascience.com/author/kevindbabitz/)\nFebruary 2, 20218 min read\n * [](https://towardsdatascience.com/an-app-to-optimize-a-pandas-dataframe-with-streamlit-759a270a6d0e/)\n## [An App to Optimize a Pandas DataFrame With Streamlit](https://towardsdatascience.com/an-app-to-optimize-a-pandas-dataframe-with-streamlit-759a270a6d0e/)\n[Programming](https://towardsdatascience.com/category/programming/)\nReduce Pandas DataFrame memory by 50% or more with this code and app.\n[Justin Chae](https://towardsdatascience.com/author/justinhchae/)\nJanuary 19, 20216 min read\n * [](https://towardsdatascience.com/build-deployable-machine-learning-pipelines-a6d7035816a6/)\n## [Build Deployable Machine Learning Pipelines](https://towardsdatascience.com/build-deployable-machine-learning-pipelines-a6d7035816a6/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nLeverage Kedro to build production-ready machine learning pipelines\n[John Adeojo](https://towardsdatascience.com/author/johnadeojo/)\nJune 30, 20238 min read\n * [](https://towardsdatascience.com/machine-learning-for-forecasting-supervised-learning-with-multivariate-time-series-b5b5044fe068/)\n## [Machine Learning for Forecasting: Supervised Learning with Multivariate Time Series](https://towardsdatascience.com/machine-learning-for-forecasting-supervised-learning-with-multivariate-time-series-b5b5044fe068/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn Introduction to the Auto-regressive Distributed Lags Model using Python.\n[Vitor Cerqueira](https://towardsdatascience.com/author/vcerq/)\nNovember 22, 20225 min read\n * [](https://towardsdatascience.com/pandas-what-ive-learned-after-my-1st-on-site-technical-interview-4fb94dbc1b45/)\n## [Pandas!!! What I've Learned after my 1st On-site Technical Interview](https://towardsdatascience.com/pandas-what-ive-learned-after-my-1st-on-site-technical-interview-4fb94dbc1b45/)\n[Programming](https://towardsdatascience.com/category/programming/)\nReflections on key text cleaning skills with Pandas str\n[Jack Chang](https://towardsdatascience.com/author/yunglinchang/)\nAugust 30, 20207 min read\n * [](https://towardsdatascience.com/why-python-loves-underscores-so-much-de03cf7bdcdd/)\n## [Why Python Loves Underscores So Much](https://towardsdatascience.com/why-python-loves-underscores-so-much-de03cf7bdcdd/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nSeveral tricks with underscores improve your Python code\n[Christopher Tao](https://towardsdatascience.com/author/christophertao/)\nJuly 4, 20217 min read\n * [](https://towardsdatascience.com/cython-for-absolute-beginners-30x-faster-code-in-two-simple-steps-bbb6c10d06ad/)\n## [Cython for absolute beginners: 30x faster code in two simple steps](https://towardsdatascience.com/cython-for-absolute-beginners-30x-faster-code-in-two-simple-steps-bbb6c10d06ad/)\n[Programming](https://towardsdatascience.com/category/programming/)\nEasy Python code compilation for blazingly fast applications\n[Mike Huls](https://towardsdatascience.com/author/mikehuls/)\nMay 22, 20223 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [  How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [  Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [  The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [  Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
"metadata": {
"title": "Fine Tuning LLMs on a Single Consumer Graphic Card | Towards Data Science",
"description": "Learnings from fine-tuning a large language model on a single consumer GPU",
"keywords": null,
"author": "Naser Tamimi",
"og:locale": "en_US",
"og:type": "article",
"og:title": "Fine Tuning LLMs on a Single Consumer Graphic Card",
"og:description": "Learnings from fine-tuning a large language model on a single consumer GPU",
"og:url": "https://towardsdatascience.com/fine-tuning-llms-on-a-single-consumer-graphic-card-6de1587daddb/",
"og:site_name": "Towards Data Science",
"og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/0y50MYOW60Csw0Ly1.png",
"og:image:width": "1024",
"og:image:height": "1024",
"og:image:type": "image/png",
"twitter:card": "summary_large_image",
"twitter:site": "@TDataScience",
"twitter:creator": "@TDataScience",
"twitter:label1": "Written by",
"twitter:data1": "Naser Tamimi",
"twitter:label2": "Est. reading time",
"twitter:data2": "10 min read",
"article:published_time": "2024-01-31T17:55:51.000Z",
"article:modified_time": "2024-01-31T17:55:51.000Z",
"article:author": "https://towardsdatascience.com/author/tamimi-naser/"
},
"paywall": {
"listed_paid": false,
"detected": false,
"marker": "",
"public_only": false,
"policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
},
"crawl": {
"success": true,
"status_code": 200,
"error": ""
},
"code_blocks": [
{
"language": "txt",
"file": "attachments/7b6a4f796cca/code/block_000.txt",
"chars": 106
},
{
"language": "txt",
"file": "attachments/7b6a4f796cca/code/block_001.txt",
"chars": 84
},
{
"language": "txt",
"file": "attachments/7b6a4f796cca/code/block_002.txt",
"chars": 177
},
{
"language": "txt",
"file": "attachments/7b6a4f796cca/code/block_003.txt",
"chars": 199
},
{
"language": "txt",
"file": "attachments/7b6a4f796cca/code/block_004.txt",
"chars": 11
},
{
"language": "txt",
"file": "attachments/7b6a4f796cca/code/block_005.txt",
"chars": 16
},
{
"language": "txt",
"file": "attachments/7b6a4f796cca/code/block_006.txt",
"chars": 1981
},
{
"language": "txt",
"file": "attachments/7b6a4f796cca/code/block_007.txt",
"chars": 1981
}
],
"images": [
{
"status": "http_404",
"url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
"src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
"alt": "",
"downloaded": false,
"discovery": "saved_markdown",
"retried_at": "2026-10-20T17:37:57Z"
},
{
"status": "http_404",
"url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
"src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
"alt": "",
"downloaded": false,
"discovery": "saved_markdown",
"retried_at": "2026-10-20T17:37:57Z"
},
{
"status": "http_404",
"url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
"src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
"alt": "",
"downloaded": false,
"discovery": "saved_markdown",
"retried_at": "2026-10-20T17:37:57Z"
},
{
"status": "http_404",
"url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
"src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
"alt": "",
"downloaded": false,
"discovery": "saved_markdown",
"retried_at": "2026-10-20T17:37:57Z"
}
],
"audio": [],
"videos": [
{
"url": "https://www.youtube.com/c/TowardsDataScience",
"kind": "channel_reference",
"status": "reference_only"
}
],
"links": [
{
"type": "internal",
"href": "https://towardsdatascience.com/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/latest/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/tag/editors-pick/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/tag/the-variable/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/shipai/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/submissions/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/category/artificial-intelligence/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/author/tamimi-naser/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/tag/artificial-intelligence/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/tag/gpu/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/tag/llm/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/tag/programming/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/tag/python/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/geospatial-analysis-in-python-and-jupyter-notebooks-f90de25b0777/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/category/data-science/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/author/franperez/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/intuition-and-implementation-of-non-max-suppression-algorithm-in-object-detection-d68ba938b630/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/author/prateekchhikara/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/what-i-learned-coding-google-from-scratch-cd7ddb02d9/"
},
{
"type": "internal",
"href": "https://towardsdatascience.com/category/programming/"
},
{
"type": "external",
"href": "https://write.insightmediagroup.io"
},
{
"type": "external",
"href": "https://www.linkedin.com/company/towards-data-science/"
},
{
"type": "external",
"href": "https://x.com/TDataScience"
},
{
"type": "external",
"href": "https://roadmap.sh/ai-engineer"
},
{
"type": "external",
"href": "https://roadmap.sh/computer-science"
},
{
"type": "external",
"href": "https://medium.com/datadriveninvestor/an-affordable-gpu-for-data-scientists-db376450117"
},
{
"type": "external",
"href": "https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/"
},
{
"type": "external",
"href": "https://github.com/Keith-Hon/bitsandbytes-windows"
},
{
"type": "external",
"href": "https://developer.nvidia.com/cuda-downloads"
},
{
"type": "external",
"href": "https://pytorch.org/"
},
{
"type": "external",
"href": "https://huggingface.co/blog/4bit-transformers-bitsandbytes"
},
{
"type": "external",
"href": "https://www.amazon.com/Generative-AWS-Context-Aware-Multimodal-Applications/dp/1098159225/ref=sr_1_1?crid=OSEFPN00F1WQ&keywords=generative+ai+on+aws&qid=1705896435&sprefix=generativ%2Caps%2C153&sr=8-1"
},
{
"type": "external",
"href": "https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md"
},
{
"type": "external",
"href": "https://arxiv.org/abs/2106.09685"
},
{
"type": "external",
"href": "https://github.com/tamiminaser/llm-single-gpu/blob/main/qlora_mistral_7b.ipynb"
},
{
"type": "external",
"href": "https://blog.ovhcloud.com/fine-tuning-llama-2-models-using-a-single-gpu-qlora-and-ai-notebooks/"
},
{
"type": "external",
"href": "https://huggingface.co/mistralai/Mistral-7B-v0.1"
},
{
"type": "external",
"href": "https://arxiv.org/abs/2310.06825"
},
{
"type": "external",
"href": "https://mistral.ai/news/announcing-mistral-7b/"
},
{
"type": "external",
"href": "https://huggingface.co/datasets/databricks/databricks-dolly-15k"
}
],
"repair_version": 2,
"repaired_at": "2026-10-20T17:23:13Z",
"source_url": "https://towardsdatascience.com/fine-tuning-llms-on-a-single-consumer-graphic-card-6de1587daddb/",
"delivery_status": "unverified",
"discovery_status": "http_403",
"missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
"repair_error": "",
"actual_video_files": [],
"expected_saved_body_images": 4,
"image_discovery": "saved_markdown",
"source_body_markdown": "\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences \nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# Fine Tuning LLMs on a Single Consumer Graphic Card\nLearnings from fine-tuning a large language model on a single consumer GPU\n[Naser Tamimi](https://towardsdatascience.com/author/tamimi-naser/)\nJanuary 31, 202410 min read\n### GENERATIVE AI\n\nImage by Author (Midjourney).\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) and [Computer Science](https://roadmap.sh/computer-science) roadmaps.\n## Background\nWhen we think about Large Language Models or any other generative models, the first hardware that comes to mind is GPU. Without GPUs, many advancements in Generative AI, machine learning, deep learning, and data science would've been impossible. If 15 years ago, gamers were enthusiastic about the latest GPU technologies, today data scientists and machine learning engineers join them and pursue the news in this field too. Although usually gamers and ML users are looking at two different kinds of GPUs and graphic cards.\nGaming users usually use consumer graphic cards (such as NVIDIA GeForce RTX Series GPUs), while ML and AI developers usually follow news about Data Center and Cloud Computing GPUs (such as V100, A100, or H100). Gaming graphic cards usually have much less GPU memory (at most 24GB as of January 2024) compared to Data Center GPUs (in the range of 40GB to 80GB usually). Also, their price is another significant difference. While most consumer graphic cards could be up to $3000, most Data Center graphic cards start from that price and can go tens of thousands of dollars easily.\nSince many people, including myself, might have a consumer graphic card for their gaming or daily use, they might be interested to see if they can use the same graphic cards for training, fine-tuning, or inference of LLM models. In 2020, I wrote a comprehensive article about whether we can use consumer graphic cards for data science projects ([link to the article](https://medium.com/datadriveninvestor/an-affordable-gpu-for-data-scientists-db376450117)). At that time, the models were mostly small ML or Deep Learning models and even a graphic card with 6GB of memory could handle many training projects. But, in this article, I am going to use such a graphic card for large language models with billions of parameters.\nFor this article, I used my GeForce 3090 RTX card which has 24GB of GPU memory. For your reference, data center graphic cards such as A100 and H100 have 40GB and 80GB of memory respectively. Also, a typical AWS EC2 p4d.24xlarge instance has 8 GPUs (V100) with a total of 320GB of GPU memory. As you can see the difference between a simple consumer GPU and a typical cloud ml instance is significant. But the question is, can we train large models on single consumer graphic cards or not? And if yes, what are the tips and learnings? Read the rest of this article, to find out.\n## Hardware and Software Setup\nBefore loading any LLM model or training dataset, we need to find out what hardware and software we need for such a process.\nAs mentioned, I used the NVIDIA GeForce RTX 3090 GPU because it has one of the highest memory (24GB) among consumer GPUs (FYI: 4090 model has the same memory size too). It is based on Ampere Architecture which is the same architecture that famous A100 GPUs have. You can see more about GeForce RTX 3090 GPU specifications [here](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/).\nAfter all the tests, I believe 24GB is the least amount of GPU memory that we need for working with LLMs with billions of parameters.\n\nImage by Author\nIn addition to the graphics card, we need to make sure that our PC has a good ventilation system. During fine-tuning, the temperature of the GPU easily goes up, and its fans are not enough to keep it cool. Higher GPU temperature can lower the GPU performance and the process will take much longer time.\nIn addition to hardware, t[here](https://github.com/Keith-Hon/bitsandbytes-windows) are some software considerations that I must mention here. First of all, if you are a Windows user, I have a piece of bad news for you. Some libraries and tools only work on Linux. Specifically, bitsandbytes which is being used frequently for model quantization is not Windows-friendly. Some people made a wrapper for Windows (for example here), but they have their cons and pros. So, my advice is either to install Linux on WSL or like me have a dual boot system and fully switch to Linux while you are working on LLMs.\nAlso, you need to install PyTorch and a compatible CUDA version. My recommendation is to install CUDA 12.3 ([link](https://developer.nvidia.com/cuda-downloads)). Then you need to go to this page, and based on your system, CUDA version, and package manager system, download and install the correct PyTorch ().\n> Note: If you are using CUDA 12.3, you might need to add or configure BNB_CUDA_VERSION and LD_LIBRARY_PATH env variables in your .bashrc file. Here is an example for your reference.\nbash\n\n```\nexport BNB_CUDA_VERSION=123\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home//local/cuda-12.3\n```\n\nAnd finally, you need to install the following packages on your system. I recommend creating a new virtual environment (venv) to avoid conflict with other installed packages on your system. Also, for your reference here are versions of packages that I successfully used:\npython\n\n```\ntorch==2.1.2\ntransformers==4.36.2\ndatasets==2.16.1\nbitsandbytes==0.42.0\npeft==0.7.1\n```\n\n## Technical Background\nNow that you have all the hardware and software ready for working with LLMs on your system, it is better to have a very brief review of the technical concepts that you will face in the next sections.\nLarge language models comprise millions or billions of parameters. Usually, we use pre-trained models that have been trained on billions and sometimes trillions of tokens after a long training process and usually millions of dollars in cost. Each of these model parameters takes 32 bits (4 bytes) of memory for loading. As a rule of thumb, each 1 billion parameters require about 4GB of memory to just load the model. One technique of using less memory for loading (and later inferencing or training a model later) is \"Quantization\". In this technique, we reduce the precision of the model weights from 32-bit full precision (fb32) to 16-bit (fp16 or bfloat16), 8-bit (int8), or even less ([read more](https://huggingface.co/blog/4bit-transformers-bitsandbytes)). As you can imagine by reducing the precision of the model weights we can load larger models into a limited memory, but this comes at the cost of reducing the model's performance. However, some studies suggested that the model performance difference between fp32 and bfloat16 is insignificant and many famous models (including Llama2) were pre-trained on bfloat16.\nQuantization is a technique that we must use when we are fine-tuning or inferencing a large language model on a single GPU with 24GB of memory. Later you will see that the `bitsandbytes` library can help us achieve model quantization.\nEven with the most rigorous quantization techniques, we are still unable to pre-train a small-size LLM model with millions of parameters. Chris Fregley et al. have a good rule of thumb in their newly published [Generative AI on AWS](https://www.amazon.com/Generative-AWS-Context-Aware-Multimodal-Applications/dp/1098159225/ref=sr_1_1?crid=OSEFPN00F1WQ&keywords=generative+ai+on+aws&qid=1705896435&sprefix=generativ%2Caps%2C153&sr=8-1) book for required memory for training a model. As they explained for each 1 billion parameters of a model, we need 6GB of memory (in 16-bit half precision) to load and train the model. Remember, the memory size is only a part of the training story. The amount of time needed to complete pre-training is another important part. As an example, the smallest Llama2 model (which is Llama2 7B) has 7 billion parameters and it took 184320 GPU hours to complete its training ([read more](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md)).\nTherefore, most people (even with significant hardware resources and budget) prefer to use a pre-trained model and only fine-tune it for their specific use case. Still, the process of full fine-tuning could be overwhelming with limited resources (such as a single GPU). Therefore, Paremer-Efficient Fine-Tuning (PEFT), which only updates a limited subset of model parameters, looks more realistic with less computed resources.\nAmong different PEFT techniques, LoRA (Low Ranking Adaption) is very popular because of its computing efficiency. In this technique, we freeze all the original model weights and instead, we train low-rank matrices that can be added to specific layers of the Transformer architecture ([read more](https://arxiv.org/abs/2106.09685)). In many cases, using LoRA for fine-tuning an LLM, we update 0.5% of model weights.\nQLoRA is a variation of LoRA that combines LoRA with the Quantization concept that we explained. Specifically, in our QLoRA implementation, we will use nf4 or Normal Float 4 for fine-tuning the model. QLoRA is very helpful in our case study of fine-tuning a large model with a single consumer GPU.\n## Coding Time\nFinally, it's coding time!\nYou can find the working Jupyter Notebook [here on my GitHub repo](https://github.com/tamiminaser/llm-single-gpu/blob/main/qlora_mistral_7b.ipynb). For many parts of this code, I got inspired and followed the instructions by Mathieu Busquet [in this neat article](https://blog.ovhcloud.com/fine-tuning-llama-2-models-using-a-single-gpu-qlora-and-ai-notebooks/).\nI will not discuss the code line by line, but I will emphasize parts of the code that are important for fine-tuning a large model on a single GPU.\n### Transformer Model\nFirst of all, I chose the [Mistral 7B model](https://huggingface.co/mistralai/Mistral-7B-v0.1) (`mistralai/Mistral-7B-v0.1`) for this testing. The Mistral 7B model developed by Mistral AI is an open-source LLM released in September 2023 ([link to the paper](https://arxiv.org/abs/2310.06825)). From many aspects, this model outperforms famous models such as Llama2 (see the following charts).\n\nImage from Mistral Release Blog Post ()\n### Dataset\nAlso, I used Databricks databricks-dolly-15k dataset (under CC BY-SA 3.0 license) for fine-tuning ([read more](https://huggingface.co/datasets/databricks/databricks-dolly-15k)). I used a small subset (1000 rows) of this data to reduce the fine-tuning time and proof of concept.\n### Configurations\nAt the model loading time, I used the following quantization config to overcome the GPU memory limitations that I am facing.\npython\n\n```\nquantization_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type=\"nf4\",\n bnb_4bit_compute_dtype=torch.bfloat16,\n)\n```\n\nThis quantization config is vital for fine-tuning a model on a single GPUT as it has one low-precision storage data type, `nf4` (4-bit Normal Float) which is a 4-bit data type, and a computation data type which is `bfloat16`. In practice, this means whenever a QLORA weight tensor is used, we dequantize the tensor to `bfloat16` and then perform matrix multiplication in 16-bit (read more in [the original paper](https://arxiv.org/abs/2305.14314)).\nAlso, as mentioned before, we are using LoRA in conjunction with Quantization, known as QLoRA, to overcome the memory limitations. Here are my configs for the LoRA:\npython\n\n```\nlora_config = LoraConfig(\n r=16,\n lora_alpha=64,\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\"], \n bias=\"none\",\n lora_dropout=0.05,\n task_type=\"CAUSAL_LM\",\n)\n```\n\nFor my LoRA configuration, I used 16 for rank. It is advised to set the rank between 4 and 16 to get a good trade-off between reducing the number of trainable parameters and the model performance. Finally, we applied LoRA to a subset of linear layers in our Mistral 7B transformer model.\n## Training and Monitoring\nUsing my single graphic card, I could complete 4 epochs of training (1000 steps). For me, one of the purposes of doing such a test, training an LLM on a single local GPU, is to monitor the hardware resources without any restrictions. One of the simplest tools for monitoring GPU during training is the Nvidia System Management Interface (SMI). Simply open a terminal and in your command line type:\nbash\n\n```\nnvidia-smi\n```\n\nor for continuously monitoring and updating, use (refreshes every 1 sec):\nbash\n\n```\nnvidia-smi -l 1\n```\n\nAs a result, you will see the memory usage by each process on your GPU. In the following SMI view, I just loaded the model and it took about 5GB of memory (thanks to the quantization). Also, as you see the model is loaded by the Anaconda3 Python (a Jupyter notebook) process.\ntext\n\n```\n+---------------------------------------------------------------------------------------+\n| NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 |\n|-----------------------------------------+----------------------+----------------------+\n| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |\n| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |\n| | | MIG M. |\n|=========================================+======================+======================|\n| 0 NVIDIA GeForce RTX 3090 On | 00000000:29:00.0 On | N/A |\n| 30% 37C P8 33W / 350W | 5346MiB / 24576MiB | 5% Default |\n| | | N/A |\n+-----------------------------------------+----------------------+----------------------+\n\n+---------------------------------------------------------------------------------------+\n| Processes: |\n| GPU GI CI PID Type Process name GPU Memory |\n| ID ID Usage |\n|=======================================================================================|\n| 0 N/A N/A 1610 G /usr/lib/xorg/Xorg 179MiB |\n| 0 N/A N/A 1820 G /usr/bin/gnome-shell 41MiB |\n| 0 N/A N/A 108004 G ...2023.3.3/host-linux-x64/nsys-ui.bin 8MiB |\n| 0 N/A N/A 168032 G ...seed-version=20240110-180219.406000 117MiB |\n| 0 N/A N/A 327503 C /home/***/anaconda3/bin/python 4880MiB |\n+---------------------------------------------------------------------------------------+\n```\n\nAnd here (the following snapshot) is the memory state after about 30 steps into the training process. As you see, the used GPU memory is now about 15GB.\ntext\n\n```\n+---------------------------------------------------------------------------------------+\n| NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 |\n|-----------------------------------------+----------------------+----------------------+\n| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |\n| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |\n| | | MIG M. |\n|=========================================+======================+======================|\n| 0 NVIDIA GeForce RTX 3090 On | 00000000:29:00.0 On | N/A |\n| 30% 57C P2 341W / 350W | 15054MiB / 24576MiB | 100% Default |\n| | | N/A |\n+-----------------------------------------+----------------------+----------------------+\n\n+---------------------------------------------------------------------------------------+\n| Processes: |\n| GPU GI CI PID Type Process name GPU Memory |\n| ID ID Usage |\n|=======================================================================================|\n| 0 N/A N/A 1610 G /usr/lib/xorg/Xorg 179MiB |\n| 0 N/A N/A 1820 G /usr/bin/gnome-shell 40MiB |\n| 0 N/A N/A 108004 G ...2023.3.3/host-linux-x64/nsys-ui.bin 8MiB |\n| 0 N/A N/A 168032 G ...seed-version=20240110-180219.406000 182MiB |\n| 0 N/A N/A 327503 C /home/***/anaconda3/bin/python 14524MiB |\n+---------------------------------------------------------------------------------------+\n```\n\nAlthough SMI is a simple tool for monitoring GPU memory usage, there are still a few advanced monitoring tools that provide more detailed information. One of these advanced tools is PyTorch Memory Snapshot which you can read more about in [this interesting article](https://pytorch.org/blog/understanding-gpu-memory-1/?utm_content=275432244&utm_medium=social&utm_source=linkedin&hss_channel=lcp-78618366).\n## Summary\nIn this article, I showed you that it is possible to fine-tune a large language model such as Mistral 7B on a single 24 GB GPU (such as NVIDIA GeForce RTX 3090 GPU). However, as discussed in detail, special PEFT techniques like QLoRA are necessary. Also, the batch size of the model is important and we might need much longer training time, just because of our limited resources.\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n\nWritten By\n## [Naser Tamimi](https://towardsdatascience.com/author/tamimi-naser/)\n[See all from Naser Tamimi](https://towardsdatascience.com/author/tamimi-naser/)\n[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Gpu](https://towardsdatascience.com/tag/gpu/)[Llm](https://towardsdatascience.com/tag/llm/)[Programming](https://towardsdatascience.com/tag/programming/)[Python](https://towardsdatascience.com/tag/python/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-llms-on-a-single-consumer-graphic-card-6de1587daddb%2F&title=Fine%20Tuning%20LLMs%20on%20a%20Single%20Consumer%20Graphic%20Card)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-llms-on-a-single-consumer-graphic-card-6de1587daddb%2F&title=Fine%20Tuning%20LLMs%20on%20a%20Single%20Consumer%20Graphic%20Card)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-llms-on-a-single-consumer-graphic-card-6de1587daddb%2F&text=Fine%20Tuning%20LLMs%20on%20a%20Single%20Consumer%20Graphic%20Card)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [](https://towardsdatascience.com/geospatial-analysis-in-python-and-jupyter-notebooks-f90de25b0777/)\n## [GeoSpatial analysis in Python and Jupyter Notebooks](https://towardsdatascience.com/geospatial-analysis-in-python-and-jupyter-notebooks-f90de25b0777/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nGeospatial analysis of Barcelona's bike rental service (bicing), using geopandas and kepler.gl.\n[Fran Pérez](https://towardsdatascience.com/author/franperez/)\nOctober 20, 20206 min read\n * [](https://towardsdatascience.com/intuition-and-implementation-of-non-max-suppression-algorithm-in-object-detection-d68ba938b630/)\n## [Intuition and Implementation of Non-Max Suppression Algorithm in Object Detection](https://towardsdatascience.com/intuition-and-implementation-of-non-max-suppression-algorithm-in-object-detection-d68ba938b630/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nThis article explains the usage and implementation of the Non-Max Suppression technique used for object detection in an image\n[Prateek Chhikara](https://towardsdatascience.com/author/prateekchhikara/)\nFebruary 1, 20225 min read\n * [](https://towardsdatascience.com/what-i-learned-coding-google-from-scratch-cd7ddb02d9/)\n## [What I Learned Coding Google From Scratch](https://towardsdatascience.com/what-i-learned-coding-google-from-scratch-cd7ddb02d9/)\n[Programming](https://towardsdatascience.com/category/programming/)\nHow to accelerate your programming development\n[Kevin Babitz](https://towardsdatascience.com/author/kevindbabitz/)\nFebruary 2, 20218 min read\n * [](https://towardsdatascience.com/an-app-to-optimize-a-pandas-dataframe-with-streamlit-759a270a6d0e/)\n## [An App to Optimize a Pandas DataFrame With Streamlit](https://towardsdatascience.com/an-app-to-optimize-a-pandas-dataframe-with-streamlit-759a270a6d0e/)\n[Programming](https://towardsdatascience.com/category/programming/)\nReduce Pandas DataFrame memory by 50% or more with this code and app.\n[Justin Chae](https://towardsdatascience.com/author/justinhchae/)\nJanuary 19, 20216 min read\n * [](https://towardsdatascience.com/build-deployable-machine-learning-pipelines-a6d7035816a6/)\n## [Build Deployable Machine Learning Pipelines](https://towardsdatascience.com/build-deployable-machine-learning-pipelines-a6d7035816a6/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nLeverage Kedro to build production-ready machine learning pipelines\n[John Adeojo](https://towardsdatascience.com/author/johnadeojo/)\nJune 30, 20238 min read\n * [](https://towardsdatascience.com/machine-learning-for-forecasting-supervised-learning-with-multivariate-time-series-b5b5044fe068/)\n## [Machine Learning for Forecasting: Supervised Learning with Multivariate Time Series](https://towardsdatascience.com/machine-learning-for-forecasting-supervised-learning-with-multivariate-time-series-b5b5044fe068/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn Introduction to the Auto-regressive Distributed Lags Model using Python.\n[Vitor Cerqueira](https://towardsdatascience.com/author/vcerq/)\nNovember 22, 20225 min read\n * [](https://towardsdatascience.com/pandas-what-ive-learned-after-my-1st-on-site-technical-interview-4fb94dbc1b45/)\n## [Pandas!!! What I've Learned after my 1st On-site Technical Interview](https://towardsdatascience.com/pandas-what-ive-learned-after-my-1st-on-site-technical-interview-4fb94dbc1b45/)\n[Programming](https://towardsdatascience.com/category/programming/)\nReflections on key text cleaning skills with Pandas str\n[Jack Chang](https://towardsdatascience.com/author/yunglinchang/)\nAugust 30, 20207 min read\n * [](https://towardsdatascience.com/why-python-loves-underscores-so-much-de03cf7bdcdd/)\n## [Why Python Loves Underscores So Much](https://towardsdatascience.com/why-python-loves-underscores-so-much-de03cf7bdcdd/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nSeveral tricks with underscores improve your Python code\n[Christopher Tao](https://towardsdatascience.com/author/christophertao/)\nJuly 4, 20217 min read\n * [](https://towardsdatascience.com/cython-for-absolute-beginners-30x-faster-code-in-two-simple-steps-bbb6c10d06ad/)\n## [Cython for absolute beginners: 30x faster code in two simple steps](https://towardsdatascience.com/cython-for-absolute-beginners-30x-faster-code-in-two-simple-steps-bbb6c10d06ad/)\n[Programming](https://towardsdatascience.com/category/programming/)\nEasy Python code compilation for blazingly fast applications\n[Mike Huls](https://towardsdatascience.com/author/mikehuls/)\nMay 22, 20223 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [  How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [  Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [  The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [  Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
"body_base_path": "attachments/7b6a4f796cca/pages/",
"body_markdown_path": "attachments/7b6a4f796cca/pages/archived-body.md",
"local_html": "attachments/7b6a4f796cca/pages/archived-body.html",
"body_source": "prior_capture"
}