![Revisit consent button](../images/body-df29beacd7496220cc94.svg) We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg) We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data. * Cookie BCTempID * Duration 10 minutes * Description No description available. * Cookie __cf_bm * Duration 1 hour * Description This cookie, set by Cloudflare, is used to support Cloudflare Bot Management. * Cookie AWSALBCORS * Duration 7 days * Description Amazon Web Services set this cookie for load balancing. * Cookie _cfuvid * Duration session * Description Cloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services * Cookie li_gc * Duration 6 months * Description Linkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes. * Cookie __hssrc * Duration session * Description This cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session. * Cookie __hssc * Duration 1 hour * Description HubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie. * Cookie wpEmojiSettingsSupports * Duration session * Description WordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly. * Cookie BCSessionID * Duration 1 year 1 month 4 days * Description Blueconic sets this cookie as a unique identifier for the BlueConic profile. * Cookie _octo * Duration 1 year * Description No description available. * Cookie logged_in * Duration 1 year * Description No description available. * Cookie __Secure-YEC * Duration past * Description YouTube sets this cookie to stores the user's video player preferences using embedded YouTube video * Cookie __eoi * Duration 6 months * Description Description is currently not available. * Cookie AWSALBTGCORS * Duration 7 days * Description No description available. * Cookie login-status-p * Duration past * Description Description is currently not available. * Cookie AWSALBTG * Duration 7 days * Description No description available. * Cookie csrf_token * Duration session * Description No description available. * Cookie token_v2 * Duration 1 day * Description Description is currently not available. * Cookie D * Duration 1 year * Description Description is currently not available. * Cookie PHPSESSID * Duration session * Description This cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed. * Cookie VISITOR_PRIVACY_METADATA * Duration 6 months * Description YouTube sets this cookie to store the user's cookie consent state for the current domain. * Cookie cookietest * Duration session * Description The cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience. * Cookie __Host-airtable-session * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie __Host-airtable-session.sig * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie m * Duration 1 year 1 month 4 days * Description Stripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly. * Cookie BIGipServer* * Duration session * Description Marketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. * Cookie __cfruid * Duration session * Description Cloudflare sets this cookie to identify trusted web traffic. * Cookie _GRECAPTCHA * Duration 6 months * Description Google Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks. * Cookie __Secure-YNID * Duration 6 months * Description Google cookie used to protect user security and prevent fraud, especially during the login process. * Cookie cookieyes-consent * Duration 1 year * Description CookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors. Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features. * Cookie lidc * Duration 1 day * Description LinkedIn sets the lidc cookie to facilitate data center selection. * Cookie brw * Duration 1 year * Description No description available. * Cookie brwConsent * Duration 5 minutes * Description Description is currently not available. * Cookie WMF-Uniq * Duration 1 year * Description Description is currently not available. * Cookie loom_anon_comment * Duration 1 year * Description No description available. * Cookie loom_referral_video * Duration session * Description Description is currently not available. * Cookie VISITOR_INFO1_LIVE * Duration 6 months * Description A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface. * Cookie yt-remote-connected-devices * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie ytidb::LAST_RESULT_ENTRY_KEY * Duration Never Expires * Description The cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future. * Cookie yt-remote-device-id * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie yt-remote-session-name * Duration session * Description The yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video. * Cookie yt-remote-fast-check-period * Duration session * Description The yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos. * Cookie yt-remote-session-app * Duration session * Description The yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player. * Cookie yt-remote-cast-available * Duration session * Description The yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player. * Cookie yt-remote-cast-installed * Duration session * Description The yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video. * Cookie cp_session * Duration 3 months * Description Codepen sets this cookie for Help systems found in the website. * Cookie loid * Duration 1 year 1 month 4 days * Description This cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform. Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc. * Cookie __hstc * Duration 6 months * Description Hubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session). * Cookie hubspotutk * Duration 6 months * Description HubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts. * Cookie _ga * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors. * Cookie _ga_* * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to store and count page views. * Cookie __Host-psifi.analyticsTrace * Duration 6 hours * Description Description is currently not available. * Cookie __Host-psifi.analyticsTraceV2 * Duration 6 hours * Description Description is currently not available. * Cookie _gh_sess * Duration session * Description GitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form. * Cookie YSC * Duration session * Description YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages. * Cookie ajs_anonymous_id * Duration 1 year * Description This cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before. * Cookie vuid * Duration 1 year 1 month 4 days * Description Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors. * Cookie AWSALB * Duration 7 days * Description AWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target. * Cookie acq * Duration past * Description Description is currently not available. * Cookie acq.sig * Duration past * Description Description is currently not available. * Cookie ptc * Duration 2 years * Description No description available. Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns. * Cookie muc_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to collect user behaviour and interaction data to optimize the website. * Cookie guest_id_marketing * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie guest_id_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie personalization_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting. * Cookie guest_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences. * Cookie bcookie * Duration 1 year * Description LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs. * Cookie __Secure-ROLLOUT_TOKEN * Duration 6 months * Description YouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment. * Cookie yt.innertube::nextId * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie yt.innertube::requests * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie session_tracker * Duration session * Description This cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website. * Cookie edgebucket * Duration session * Description Reddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content. * Cookie did * Duration 1 year * Description Arbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year. Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation [](https://towardsdatascience.com/) Publish AI, ML & data-science insights to a global community of data professionals. Sign in[Submit an Article](https://write.insightmediagroup.io) [Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/) * * * [Write For TDS](https://towardsdatascience.com/submissions/) [LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) # Fine Tuning LLMs on a Single Consumer Graphic Card Learnings from fine-tuning a large language model on a single consumer GPU [Naser Tamimi](https://towardsdatascience.com/author/tamimi-naser/) January 31, 202410 min read ### GENERATIVE AI ![Image by Author \(Midjourney\).](../images/000.png) Image by Author (Midjourney). Learn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) and [Computer Science](https://roadmap.sh/computer-science) roadmaps. ## Background When we think about Large Language Models or any other generative models, the first hardware that comes to mind is GPU. Without GPUs, many advancements in Generative AI, machine learning, deep learning, and data science would've been impossible. If 15 years ago, gamers were enthusiastic about the latest GPU technologies, today data scientists and machine learning engineers join them and pursue the news in this field too. Although usually gamers and ML users are looking at two different kinds of GPUs and graphic cards. Gaming users usually use consumer graphic cards (such as NVIDIA GeForce RTX Series GPUs), while ML and AI developers usually follow news about Data Center and Cloud Computing GPUs (such as V100, A100, or H100). Gaming graphic cards usually have much less GPU memory (at most 24GB as of January 2024) compared to Data Center GPUs (in the range of 40GB to 80GB usually). Also, their price is another significant difference. While most consumer graphic cards could be up to $3000, most Data Center graphic cards start from that price and can go tens of thousands of dollars easily. Since many people, including myself, might have a consumer graphic card for their gaming or daily use, they might be interested to see if they can use the same graphic cards for training, fine-tuning, or inference of LLM models. In 2020, I wrote a comprehensive article about whether we can use consumer graphic cards for data science projects ([link to the article](https://medium.com/datadriveninvestor/an-affordable-gpu-for-data-scientists-db376450117)). At that time, the models were mostly small ML or Deep Learning models and even a graphic card with 6GB of memory could handle many training projects. But, in this article, I am going to use such a graphic card for large language models with billions of parameters. For this article, I used my GeForce 3090 RTX card which has 24GB of GPU memory. For your reference, data center graphic cards such as A100 and H100 have 40GB and 80GB of memory respectively. Also, a typical AWS EC2 p4d.24xlarge instance has 8 GPUs (V100) with a total of 320GB of GPU memory. As you can see the difference between a simple consumer GPU and a typical cloud ml instance is significant. But the question is, can we train large models on single consumer graphic cards or not? And if yes, what are the tips and learnings? Read the rest of this article, to find out. ## Hardware and Software Setup Before loading any LLM model or training dataset, we need to find out what hardware and software we need for such a process. As mentioned, I used the NVIDIA GeForce RTX 3090 GPU because it has one of the highest memory (24GB) among consumer GPUs (FYI: 4090 model has the same memory size too). It is based on Ampere Architecture which is the same architecture that famous A100 GPUs have. You can see more about GeForce RTX 3090 GPU specifications [here](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/). After all the tests, I believe 24GB is the least amount of GPU memory that we need for working with LLMs with billions of parameters. ![Image by Author](../images/001.jpeg) Image by Author In addition to the graphics card, we need to make sure that our PC has a good ventilation system. During fine-tuning, the temperature of the GPU easily goes up, and its fans are not enough to keep it cool. Higher GPU temperature can lower the GPU performance and the process will take much longer time. In addition to hardware, t[here](https://github.com/Keith-Hon/bitsandbytes-windows) are some software considerations that I must mention here. First of all, if you are a Windows user, I have a piece of bad news for you. Some libraries and tools only work on Linux. Specifically, bitsandbytes which is being used frequently for model quantization is not Windows-friendly. Some people made a wrapper for Windows (for example here), but they have their cons and pros. So, my advice is either to install Linux on WSL or like me have a dual boot system and fully switch to Linux while you are working on LLMs. Also, you need to install PyTorch and a compatible CUDA version. My recommendation is to install CUDA 12.3 ([link](https://developer.nvidia.com/cuda-downloads)). Then you need to go to this page, and based on your system, CUDA version, and package manager system, download and install the correct PyTorch (). > Note: If you are using CUDA 12.3, you might need to add or configure BNB_CUDA_VERSION and LD_LIBRARY_PATH env variables in your .bashrc file. Here is an example for your reference. bash ``` export BNB_CUDA_VERSION=123 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home//local/cuda-12.3 ``` And finally, you need to install the following packages on your system. I recommend creating a new virtual environment (venv) to avoid conflict with other installed packages on your system. Also, for your reference here are versions of packages that I successfully used: python ``` torch==2.1.2 transformers==4.36.2 datasets==2.16.1 bitsandbytes==0.42.0 peft==0.7.1 ``` ## Technical Background Now that you have all the hardware and software ready for working with LLMs on your system, it is better to have a very brief review of the technical concepts that you will face in the next sections. Large language models comprise millions or billions of parameters. Usually, we use pre-trained models that have been trained on billions and sometimes trillions of tokens after a long training process and usually millions of dollars in cost. Each of these model parameters takes 32 bits (4 bytes) of memory for loading. As a rule of thumb, each 1 billion parameters require about 4GB of memory to just load the model. One technique of using less memory for loading (and later inferencing or training a model later) is "Quantization". In this technique, we reduce the precision of the model weights from 32-bit full precision (fb32) to 16-bit (fp16 or bfloat16), 8-bit (int8), or even less ([read more](https://huggingface.co/blog/4bit-transformers-bitsandbytes)). As you can imagine by reducing the precision of the model weights we can load larger models into a limited memory, but this comes at the cost of reducing the model's performance. However, some studies suggested that the model performance difference between fp32 and bfloat16 is insignificant and many famous models (including Llama2) were pre-trained on bfloat16. Quantization is a technique that we must use when we are fine-tuning or inferencing a large language model on a single GPU with 24GB of memory. Later you will see that the `bitsandbytes` library can help us achieve model quantization. Even with the most rigorous quantization techniques, we are still unable to pre-train a small-size LLM model with millions of parameters. Chris Fregley et al. have a good rule of thumb in their newly published [Generative AI on AWS](https://www.amazon.com/Generative-AWS-Context-Aware-Multimodal-Applications/dp/1098159225/ref=sr_1_1?crid=OSEFPN00F1WQ&keywords=generative+ai+on+aws&qid=1705896435&sprefix=generativ%2Caps%2C153&sr=8-1) book for required memory for training a model. As they explained for each 1 billion parameters of a model, we need 6GB of memory (in 16-bit half precision) to load and train the model. Remember, the memory size is only a part of the training story. The amount of time needed to complete pre-training is another important part. As an example, the smallest Llama2 model (which is Llama2 7B) has 7 billion parameters and it took 184320 GPU hours to complete its training ([read more](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md)). Therefore, most people (even with significant hardware resources and budget) prefer to use a pre-trained model and only fine-tune it for their specific use case. Still, the process of full fine-tuning could be overwhelming with limited resources (such as a single GPU). Therefore, Paremer-Efficient Fine-Tuning (PEFT), which only updates a limited subset of model parameters, looks more realistic with less computed resources. Among different PEFT techniques, LoRA (Low Ranking Adaption) is very popular because of its computing efficiency. In this technique, we freeze all the original model weights and instead, we train low-rank matrices that can be added to specific layers of the Transformer architecture ([read more](https://arxiv.org/abs/2106.09685)). In many cases, using LoRA for fine-tuning an LLM, we update 0.5% of model weights. QLoRA is a variation of LoRA that combines LoRA with the Quantization concept that we explained. Specifically, in our QLoRA implementation, we will use nf4 or Normal Float 4 for fine-tuning the model. QLoRA is very helpful in our case study of fine-tuning a large model with a single consumer GPU. ## Coding Time Finally, it's coding time! You can find the working Jupyter Notebook [here on my GitHub repo](https://github.com/tamiminaser/llm-single-gpu/blob/main/qlora_mistral_7b.ipynb). For many parts of this code, I got inspired and followed the instructions by Mathieu Busquet [in this neat article](https://blog.ovhcloud.com/fine-tuning-llama-2-models-using-a-single-gpu-qlora-and-ai-notebooks/). I will not discuss the code line by line, but I will emphasize parts of the code that are important for fine-tuning a large model on a single GPU. ### Transformer Model First of all, I chose the [Mistral 7B model](https://huggingface.co/mistralai/Mistral-7B-v0.1) (`mistralai/Mistral-7B-v0.1`) for this testing. The Mistral 7B model developed by Mistral AI is an open-source LLM released in September 2023 ([link to the paper](https://arxiv.org/abs/2310.06825)). From many aspects, this model outperforms famous models such as Llama2 (see the following charts). ![Image from Mistral Release Blog Post \(https://mistral.ai/news/announcing-mistral-7b/\)](../images/002.png) Image from Mistral Release Blog Post () ### Dataset Also, I used Databricks databricks-dolly-15k dataset (under CC BY-SA 3.0 license) for fine-tuning ([read more](https://huggingface.co/datasets/databricks/databricks-dolly-15k)). I used a small subset (1000 rows) of this data to reduce the fine-tuning time and proof of concept. ### Configurations At the model loading time, I used the following quantization config to overcome the GPU memory limitations that I am facing. python ``` quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, ) ``` This quantization config is vital for fine-tuning a model on a single GPUT as it has one low-precision storage data type, `nf4` (4-bit Normal Float) which is a 4-bit data type, and a computation data type which is `bfloat16`. In practice, this means whenever a QLORA weight tensor is used, we dequantize the tensor to `bfloat16` and then perform matrix multiplication in 16-bit (read more in [the original paper](https://arxiv.org/abs/2305.14314)). Also, as mentioned before, we are using LoRA in conjunction with Quantization, known as QLoRA, to overcome the memory limitations. Here are my configs for the LoRA: python ``` lora_config = LoraConfig( r=16, lora_alpha=64, target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"], bias="none", lora_dropout=0.05, task_type="CAUSAL_LM", ) ``` For my LoRA configuration, I used 16 for rank. It is advised to set the rank between 4 and 16 to get a good trade-off between reducing the number of trainable parameters and the model performance. Finally, we applied LoRA to a subset of linear layers in our Mistral 7B transformer model. ## Training and Monitoring Using my single graphic card, I could complete 4 epochs of training (1000 steps). For me, one of the purposes of doing such a test, training an LLM on a single local GPU, is to monitor the hardware resources without any restrictions. One of the simplest tools for monitoring GPU during training is the Nvidia System Management Interface (SMI). Simply open a terminal and in your command line type: bash ``` nvidia-smi ``` or for continuously monitoring and updating, use (refreshes every 1 sec): bash ``` nvidia-smi -l 1 ``` As a result, you will see the memory usage by each process on your GPU. In the following SMI view, I just loaded the model and it took about 5GB of memory (thanks to the quantization). Also, as you see the model is loaded by the Anaconda3 Python (a Jupyter notebook) process. text ``` +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce RTX 3090 On | 00000000:29:00.0 On | N/A | | 30% 37C P8 33W / 350W | 5346MiB / 24576MiB | 5% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 1610 G /usr/lib/xorg/Xorg 179MiB | | 0 N/A N/A 1820 G /usr/bin/gnome-shell 41MiB | | 0 N/A N/A 108004 G ...2023.3.3/host-linux-x64/nsys-ui.bin 8MiB | | 0 N/A N/A 168032 G ...seed-version=20240110-180219.406000 117MiB | | 0 N/A N/A 327503 C /home/***/anaconda3/bin/python 4880MiB | +---------------------------------------------------------------------------------------+ ``` And here (the following snapshot) is the memory state after about 30 steps into the training process. As you see, the used GPU memory is now about 15GB. text ``` +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce RTX 3090 On | 00000000:29:00.0 On | N/A | | 30% 57C P2 341W / 350W | 15054MiB / 24576MiB | 100% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 1610 G /usr/lib/xorg/Xorg 179MiB | | 0 N/A N/A 1820 G /usr/bin/gnome-shell 40MiB | | 0 N/A N/A 108004 G ...2023.3.3/host-linux-x64/nsys-ui.bin 8MiB | | 0 N/A N/A 168032 G ...seed-version=20240110-180219.406000 182MiB | | 0 N/A N/A 327503 C /home/***/anaconda3/bin/python 14524MiB | +---------------------------------------------------------------------------------------+ ``` Although SMI is a simple tool for monitoring GPU memory usage, there are still a few advanced monitoring tools that provide more detailed information. One of these advanced tools is PyTorch Memory Snapshot which you can read more about in [this interesting article](https://pytorch.org/blog/understanding-gpu-memory-1/?utm_content=275432244&utm_medium=social&utm_source=linkedin&hss_channel=lcp-78618366). ## Summary In this article, I showed you that it is possible to fine-tune a large language model such as Mistral 7B on a single 24 GB GPU (such as NVIDIA GeForce RTX 3090 GPU). However, as discussed in detail, special PEFT techniques like QLoRA are necessary. Also, the batch size of the model is important and we might need much longer training time, just because of our limited resources. ## Register Now to Continue Reading ### Unlock Towards Data Science with a Free Account All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account * * * ![](../images/body-909ce5c2500185af5a85.jpeg) Written By ## [Naser Tamimi](https://towardsdatascience.com/author/tamimi-naser/) [See all from Naser Tamimi](https://towardsdatascience.com/author/tamimi-naser/) [Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Gpu](https://towardsdatascience.com/tag/gpu/)[Llm](https://towardsdatascience.com/tag/llm/)[Programming](https://towardsdatascience.com/tag/programming/)[Python](https://towardsdatascience.com/tag/python/) Share This Article * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-llms-on-a-single-consumer-graphic-card-6de1587daddb%2F&title=Fine%20Tuning%20LLMs%20on%20a%20Single%20Consumer%20Graphic%20Card) * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-llms-on-a-single-consumer-graphic-card-6de1587daddb%2F&title=Fine%20Tuning%20LLMs%20on%20a%20Single%20Consumer%20Graphic%20Card) * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-llms-on-a-single-consumer-graphic-card-6de1587daddb%2F&text=Fine%20Tuning%20LLMs%20on%20a%20Single%20Consumer%20Graphic%20Card) Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. [Write for TDS](https://towardsdatascience.com/submissions/) ## Related Articles * [![](../images/003.png)](https://towardsdatascience.com/geospatial-analysis-in-python-and-jupyter-notebooks-f90de25b0777/) ## [GeoSpatial analysis in Python and Jupyter Notebooks](https://towardsdatascience.com/geospatial-analysis-in-python-and-jupyter-notebooks-f90de25b0777/) [Data Science](https://towardsdatascience.com/category/data-science/) Geospatial analysis of Barcelona's bike rental service (bicing), using geopandas and kepler.gl. [Fran Pérez](https://towardsdatascience.com/author/franperez/) October 20, 20206 min read * [![](../images/004.png)](https://towardsdatascience.com/intuition-and-implementation-of-non-max-suppression-algorithm-in-object-detection-d68ba938b630/) ## [Intuition and Implementation of Non-Max Suppression Algorithm in Object Detection](https://towardsdatascience.com/intuition-and-implementation-of-non-max-suppression-algorithm-in-object-detection-d68ba938b630/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) This article explains the usage and implementation of the Non-Max Suppression technique used for object detection in an image [Prateek Chhikara](https://towardsdatascience.com/author/prateekchhikara/) February 1, 20225 min read * [![](../images/005.jpg)](https://towardsdatascience.com/what-i-learned-coding-google-from-scratch-cd7ddb02d9/) ## [What I Learned Coding Google From Scratch](https://towardsdatascience.com/what-i-learned-coding-google-from-scratch-cd7ddb02d9/) [Programming](https://towardsdatascience.com/category/programming/) How to accelerate your programming development [Kevin Babitz](https://towardsdatascience.com/author/kevindbabitz/) February 2, 20218 min read * [![](../images/006.jpg)](https://towardsdatascience.com/an-app-to-optimize-a-pandas-dataframe-with-streamlit-759a270a6d0e/) ## [An App to Optimize a Pandas DataFrame With Streamlit](https://towardsdatascience.com/an-app-to-optimize-a-pandas-dataframe-with-streamlit-759a270a6d0e/) [Programming](https://towardsdatascience.com/category/programming/) Reduce Pandas DataFrame memory by 50% or more with this code and app. [Justin Chae](https://towardsdatascience.com/author/justinhchae/) January 19, 20216 min read * [![](../images/007.png)](https://towardsdatascience.com/build-deployable-machine-learning-pipelines-a6d7035816a6/) ## [Build Deployable Machine Learning Pipelines](https://towardsdatascience.com/build-deployable-machine-learning-pipelines-a6d7035816a6/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Leverage Kedro to build production-ready machine learning pipelines [John Adeojo](https://towardsdatascience.com/author/johnadeojo/) June 30, 20238 min read * [![](../images/008.png)](https://towardsdatascience.com/machine-learning-for-forecasting-supervised-learning-with-multivariate-time-series-b5b5044fe068/) ## [Machine Learning for Forecasting: Supervised Learning with Multivariate Time Series](https://towardsdatascience.com/machine-learning-for-forecasting-supervised-learning-with-multivariate-time-series-b5b5044fe068/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) An Introduction to the Auto-regressive Distributed Lags Model using Python. [Vitor Cerqueira](https://towardsdatascience.com/author/vcerq/) November 22, 20225 min read * [![](../images/009.jpeg)](https://towardsdatascience.com/pandas-what-ive-learned-after-my-1st-on-site-technical-interview-4fb94dbc1b45/) ## [Pandas!!! What I've Learned after my 1st On-site Technical Interview](https://towardsdatascience.com/pandas-what-ive-learned-after-my-1st-on-site-technical-interview-4fb94dbc1b45/) [Programming](https://towardsdatascience.com/category/programming/) Reflections on key text cleaning skills with Pandas str [Jack Chang](https://towardsdatascience.com/author/yunglinchang/) August 30, 20207 min read * [![](../images/010.jpeg)](https://towardsdatascience.com/why-python-loves-underscores-so-much-de03cf7bdcdd/) ## [Why Python Loves Underscores So Much](https://towardsdatascience.com/why-python-loves-underscores-so-much-de03cf7bdcdd/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) Several tricks with underscores improve your Python code [Christopher Tao](https://towardsdatascience.com/author/christophertao/) July 4, 20217 min read * [![](../images/011.jpg)](https://towardsdatascience.com/cython-for-absolute-beginners-30x-faster-code-in-two-simple-steps-bbb6c10d06ad/) ## [Cython for absolute beginners: 30x faster code in two simple steps](https://towardsdatascience.com/cython-for-absolute-beginners-30x-faster-code-in-two-simple-steps-bbb6c10d06ad/) [Programming](https://towardsdatascience.com/category/programming/) Easy Python code compilation for blazingly fast applications [Mike Huls](https://towardsdatascience.com/author/mikehuls/) May 22, 20223 min read * [YouTube](https://www.youtube.com/c/TowardsDataScience) * [X](https://x.com/TDataScience) * [LinkedIn](https://www.linkedin.com/company/towards-data-science/) * [Threads](https://www.threads.net/@towardsdatascience) * [Bluesky](https://bsky.app/profile/towardsdatascience.com) [](https://towardsdatascience.com/) Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/) © Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network) #### Recommended Articles Close * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ "How Relevance Models Foreshadowed Transformers for NLP") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ "Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ "The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ "Does Your Company Have a Data Strategy?")