{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Philipp Schmid",
  "title_listed": "Fine-tune Multimodal LLMs with TRL",
  "keywords": "多模态,微调,TRL",
  "domain": "图像处理",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://www.philschmid.de/fine-tune-multimodal-llms-with-trl",
  "publish_date": "2024-09",
  "corpus_stage": "预训练/SFT",
  "notes": "用TRL微调多模态LLM",
  "id": "3c43db3184f5",
  "collected_at": "2026-10-02T14:29:19",
  "title": "How to Fine-Tune Multimodal Models or VLMs with Hugging Face TRL",
  "body_markdown": "How to Fine-Tune Multimodal Models or VLMs with Hugging Face TRL\nSeptember 30, 2024\n14\nminute read\nView Code\nMultimodal LLMs are making tremendous progress recently. We now have a diverse ecosystem of powerful open Multimodal models, mostly Vision-Language Models (VLM), including Meta AI's\nLlama-3.2-11B-Vision\n, Mistral AI's\nPixtral-12B\n, Qwen's\nQwen2-VL-7B\n, and Allen AI's\nMolmo-7B-D-0924\n.\nThese VLMs can handle a variety of multimodal tasks, including image captioning, visual question answering, and image-text matching without additional training. However, to customize a model for your specific application, you may need to fine-tune it on your data to achieve higher quality results or to create a more efficient model for your use case.\nThis blog post walks you through how to fine-tune open VLMs using Hugging Face\nTRL\n,\nTransformers\n&\ndatasets\nin 2024. We'll cover:\nDefine our multimodal use case\nSetup development environment\nCreate and prepare the multimodal dataset\nFine-tune VLM using\ntrl\nand the\nSFTTrainer\nTest and evaluate the VLM\nNote: This blog was created to run on consumer size GPUs (24GB), e.g. NVIDIA A10G or RTX 4090/3090, but can be easily adapted to run on bigger GPUs.\n1. Define our multimodal use case\nWhen fine-tuning VLMs, it's crucial to clearly define your use case and the multimodal task you want to solve. This will guide your choice of base model and help you create an appropriate dataset for fine-tuning. If you haven't defined your use case yet, you might want to revisit your requirements.\nIt's worth noting that for most use cases fine-tuning might not be the first option. We recommend evaluating pre-trained models or API-based solutions before committing to fine-tuning your own model.\nAs an example, we'll use the following multimodal use case:\nWe want to fine-tune a model that can generate detailed product descriptions based on product images and basic metadata. This model will be integrated into our e-commerce platform to help sellers create more compelling listings. The goal is to reduce the time it takes to create product descriptions and improve their quality and consistency.\nExisting models might already be very good for this use case, but you might want to tweak/tune it to your specific needs. This image-to-text generation task is well-suited for fine-tuning VLMs, as it requires understanding visual features and combining them with textual information to produce coherent and relevant descriptions. I created a test dataset for this use case using Gemini 1.5\nphilschmid/amazon-product-descriptions-vlm\n.\n2. Setup development environment\nOur first step is to install Hugging Face Libraries and Pyroch, including trl, transformers and datasets. If you haven't heard of trl yet, don't worry. It is a library on top of transformers and datasets, which makes it easier to fine-tune, rlhf, align open LLMs.\nPython\n# Install Pytorch & other libraries\n%pip install\n\"torch==2.4.0\"\ntensorboard pillow\n# Install Hugging Face libraries\n%pip install\n--\nupgrade \\\n\"transformers==4.45.1\"\n\\\n\"datasets==3.0.1\"\n\\\n\"accelerate==0.34.2\"\n\\\n\"evaluate==0.4.3\"\n\\\n\"bitsandbytes==0.44.0\"\n\\\n\"trl==0.11.1\"\n\\\n\"peft==0.13.0\"\n\\\n\"qwen-vl-utils\"\nWe will use the\nHugging Face Hub\nas a remote model versioning service. This means we will automatically push our model, logs and information to the Hub during training. You must register on the\nHugging Face\nfor this. After you have an account, we will use the\nlogin\nutil from the\nhuggingface_hub\npackage to log into our account and store our token (access key) on the disk.\nPython\nfrom\nhuggingface_hub\nimport\nlogin\nlogin\n(\ntoken\n=\n\"\"\n,\n# ADD YOUR TOKEN HERE\nadd_to_git_credential\n=\nTrue\n)\n3. Create and prepare the dataset\nOnce you have determined that fine-tuning is the right solution we need to create a dataset to fine-tune our model. We have to prepare the dataset in a format that the model can understand.\nIn our example we will use\nphilschmid/amazon-product-descriptions-vlm\n, which contains 1,350 amazon products with title, images and descriptions and metadata. We want to fine-tune our model to generate product descriptions based on the images, title and metadata. Therefore we need to create a prompt including the title, metadata and image and the completion is the description.\nTRL supports popular instruction and conversation dataset formats. This means we only need to convert our dataset to one of the supported formats and\ntrl\nwill take care of the rest.\nJSON\n\"messages\"\n: [\n{\n\"role\"\n:\n\"system\"\n,\n\"content\"\n: [{\n\"type\"\n:\n\"text\"\n,\n\"text\"\n:\n\"You are a helpful....\"\n}]},\n{\n\"role\"\n:\n\"user\"\n,\n\"content\"\n: [{\n\"type\"\n:\n\"text\"\n,\n\"text\"\n:\n\"How many dogs are in the image?\"\n,\n\"type\"\n:\n\"image\"\n,\n\"text\"\n:\n<PIL.Image>\n}]},\n{\n\"role\"\n:\n\"assistant\"\n,\n\"content\"\n: [{\n\"type\"\n:\n\"text\"\n,\n\"text\"\n:\n\"There are 3 dogs in the image.\"\n}]}\n],\nIn our example we are going to load our dataset using the Datasets library and apply our frompt and convert it into the the conversational format.\nLets start with defining our instruction prompt.\nPython\n# note the image is not provided in the prompt its included as part of the \"processor\"\nprompt=\n\"\"\"Create a Short Product description based on the provided ##PRODUCT NAME## and ##CATEGORY## and image.\nOnly return description. The description should be SEO optimized and for a better mobile search experience.\n##PRODUCT NAME##: {product_name}\n##CATEGORY##: {category}\"\"\"\nsystem_message =\n\"You are an expert product description writer for Amazon.\"\nNow, we can format our dataset.\nPython\nfrom\ndatasets\nimport\nload_dataset\n# Convert dataset to OAI messages\ndef\nformat_data\n(\nsample\n):\nreturn\n{\n\"messages\"\n: [\n{\n\"role\"\n:\n\"system\"\n,\n\"content\"\n: [{\n\"type\"\n:\n\"text\"\n,\n\"text\"\n: system_message}],\n},\n{\n\"role\"\n:\n\"user\"\n,\n\"content\"\n: [\n{\n\"type\"\n:\n\"text\"\n,\n\"text\"\n: prompt.\nformat\n(\nproduct_name\n=sample[\n\"Product Name\"\n],\ncategory\n=sample[\n\"Category\"\n]),\n},{\n\"type\"\n:\n\"image\"\n,\n\"image\"\n: sample[\n\"image\"\n],\n}\n],\n},\n{\n\"role\"\n:\n\"assistant\"\n,\n\"content\"\n: [{\n\"type\"\n:\n\"text\"\n,\n\"text\"\n: sample[\n\"description\"\n]}],\n},\n],\n}\n# Load dataset from the hub\ndataset_id =\n\"philschmid/amazon-product-descriptions-vlm\"\ndataset =\nload_dataset\n(\n\"philschmid/amazon-product-descriptions-vlm\"\n,\nsplit\n=\n\"train\"\n)\n# Convert dataset to OAI messages\n# need to use list comprehension to keep Pil.Image type, .mape convert image to bytes\ndataset = [\nformat_data\n(sample)\nfor\nsample\nin\ndataset]\nprint(dataset[\n345\n][\n\"messages\"\n])\n4. Fine-tune VLM using\ntrl\nand the\nSFTTrainer\nWe are now ready to fine-tune our model. We will use the\nSFTTrainer\nfrom\ntrl\nto fine-tune our model. The\nSFTTrainer\nmakes it straightfoward to supervise fine-tune open LLMs and VLMs. The\nSFTTrainer\nis a subclass of the\nTrainer\nfrom the\ntransformers\nlibrary and supports all the same features, including logging, evaluation, and checkpointing, but adds additiional quality of life features.\nWe will use the PEFT features in our example. As peft method we will use\nQLoRA\na technique to reduce the memory footprint of large language models during finetuning, without sacrificing performance by using quantization. If you want to learn more about QLoRA and how it works, check out\nMaking LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA\nblog post.\nNote: We cannot use Flash Attention as we need to pad our multimodal inputs.\nWe are going to use Qwen 2 VL 7B model, but we can easily swap out the model for another model, including Meta AI's\nLlama-3.2-11B-Vision\n, Mistral AI's\nPixtral-12B\nor any other LLMs by changing our\nmodel_id\nvariable. We will use bitsandbytes to quantize our model to 4-bit.\nNote: Be aware the bigger the model the more memory it will require. In our example we will use the 7B version, which can be tuned on 24GB GPUs.\nCorrectly, preparing the LLM, Tokenizer and Processor for training VLMs is crucial. The Processor is responsible for including the special tokens and image features in the input.\nPython\nimport\ntorch\nfrom\ntransformers\nimport\nAutoModelForVision2Seq, AutoProcessor, BitsAndBytesConfig\n# Hugging Face model id\nmodel_id =\n\"Qwen/Qwen2-VL-7B-Instruct\"\n# BitsAndBytesConfig int-4 config\nbnb_config =\nBitsAndBytesConfig\n(\nload_in_4bit\n=\nTrue\n,\nbnb_4bit_use_double_quant\n=\nTrue\n,\nbnb_4bit_quant_type\n=\n\"nf4\"\n,\nbnb_4bit_compute_dtype\n=torch.bfloat16\n)\n# Load model and tokenizer\nmodel = AutoModelForVision2Seq.\nfrom_pretrained\n(\nmodel_id,\ndevice_map\n=\n\"auto\"\n,\n# attn_implementation=\"flash_attention_2\", # not supported for training\ntorch_dtype\n=torch.bfloat16,\nquantization_config\n=bnb_config\n)\nprocessor = AutoProcessor.\nfrom_pretrained\n(model_id)\nPython\n# Preparation for inference\ntext = processor.\napply_chat_template\n(\ndataset[\n2\n][\n\"messages\"\n],\ntokenize\n=\nFalse\n,\nadd_generation_prompt\n=\nFalse\n)\nprint(text)\nThe\nSFTTrainer\nsupports a native integration with\npeft\n, which makes it super easy to efficiently tune LLMs using, e.g. QLoRA. We only need to create our\nLoraConfig\nand provide it to the trainer. Our\nLoraConfig\nparameters are defined based on the\nqlora paper\nand sebastian's\nblog post\n.\nPython\nfrom\npeft\nimport\nLoraConfig\n# LoRA config based on QLoRA paper & Sebastian Raschka experiment\npeft_config =\nLoraConfig\n(\nlora_alpha\n=\n16\n,\nlora_dropout\n=\n0.05\n,\nr\n=\n8\n,\nbias\n=\n\"none\"\n,\ntarget_modules\n=[\n\"q_proj\"\n,\n\"v_proj\"\n],\ntask_type\n=\n\"CAUSAL_LM\"\n,\n)\nBefore we can start our training we need to define the hyperparameters (\nSFTConfig\n) we want to use and make sure our inputs are correcty provided to the model. Different to text-only supervised fine-tuning we need to provide the image to the model as well. Therefore we create a custom\nDataCollator\nwhich formates the inputs correctly and include the image features. We use the\nprocess_vision_info\nmethod from a utility package the Qwen2 team provides. If you are using another model, e.g. Llama 3.2 Vision you might have to check if that creates the same processsed image information.\nPython\nfrom\ntrl\nimport\nSFTConfig\nfrom\ntransformers\nimport\nQwen2VLProcessor\nfrom\nqwen_vl_utils\nimport\nprocess_vision_info\nargs =\nSFTConfig\n(\noutput_dir\n=\n\"qwen2-7b-instruct-amazon-description\"\n,\n# directory to save and repository id\nnum_train_epochs\n=\n3\n,\n# number of training epochs\nper_device_train_batch_size\n=\n4\n,\n# batch size per device during training\ngradient_accumulation_steps\n=\n8\n,\n# number of steps before performing a backward/update pass\ngradient_checkpointing\n=\nTrue\n,\n# use gradient checkpointing to save memory\noptim\n=\n\"adamw_torch_fused\"\n,\n# use fused adamw optimizer\nlogging_steps\n=\n5\n,\n# log every 10 steps\nsave_strategy\n=\n\"epoch\"\n,\n# save checkpoint every epoch\nlearning_rate\n=\n2e-4\n,\n# learning rate, based on QLoRA paper\nbf16\n=\nTrue\n,\n# use bfloat16 precision\ntf32\n=\nTrue\n,\n# use tf32 precision\nmax_grad_norm\n=\n0.3\n,\n# max gradient norm based on QLoRA paper\nwarmup_ratio\n=\n0.03\n,\n# warmup ratio based on QLoRA paper\nlr_scheduler_type\n=\n\"constant\"\n,\n# use constant learning rate scheduler\npush_to_hub\n=\nTrue\n,\n# push model to hub\nreport_to\n=\n\"tensorboard\"\n,\n# report metrics to tensorboard\ngradient_checkpointing_kwargs\n= {\n\"use_reentrant\"\n:\nFalse\n},\n# use reentrant checkpointing\ndataset_text_field\n=\n\"\"\n,\n# need a dummy field for collator\ndataset_kwargs\n= {\n\"skip_prepare_dataset\"\n:\nTrue\n}\n# important for collator\n)\nargs.remove_unused_columns=\nFalse\n# Create a data collator to encode text and image pairs\ndef\ncollate_fn\n(\nexamples\n):\n# Get the texts and images, and apply the chat template\ntexts = [processor.\napply_chat_template\n(example[\n\"messages\"\n],\ntokenize\n=\nFalse\n)\nfor\nexample\nin\nexamples]\nimage_inputs = [\nprocess_vision_info\n(example[\n\"messages\"\n])[\n0\n]\nfor\nexample\nin\nexamples]\n# Tokenize the texts and process the images\nbatch =\nprocessor\n(\ntext\n=texts,\nimages\n=image_inputs,\nreturn_tensors\n=\n\"pt\"\n,\npadding\n=\nTrue\n)\n# The labels are the input_ids, and we mask the padding tokens in the loss computation\nlabels = batch[\n\"input_ids\"\n].\nclone\n()\nlabels[labels == processor.tokenizer.pad_token_id] = -\n100\n#\n# Ignore the image token index in the loss computation (model specific)\nif\nisinstance(processor, Qwen2VLProcessor):\nimage_tokens = [\n151652\n,\n151653\n,\n151655\n]\nelse\n:\nimage_tokens = [processor.tokenizer.\nconvert_tokens_to_ids\n(processor.image_token)]\nfor\nimage_token_id\nin\nimage_tokens:\nlabels[labels == image_token_id] = -\n100\nbatch[\n\"labels\"\n] = labels\nreturn\nbatch\nWe now have every building block we need to create our\nSFTTrainer\nto start then training our model.\nPython\nfrom\ntrl\nimport\nSFTTrainer\ntrainer =\nSFTTrainer\n(\nmodel\n=model,\nargs\n=args,\ntrain_dataset\n=dataset,\ndata_collator\n=collate_fn,\ndataset_text_field\n=\n\"\"\n,\n# needs dummy value\npeft_config\n=peft_config,\ntokenizer\n=processor.tokenizer,\n)\nStart training our model by calling the\ntrain()\nmethod on our\nTrainer\ninstance. This will start the training loop and train our model for 3 epochs. Since we are using a PEFT method, we will only save the adapted model weights and not the full model.\nPython\n# start training, the model will be automatically saved to the hub and the output directory\ntrainer.\ntrain\n()\n# save model\ntrainer.\nsave_model\n(args.output_dir)\nTraining for 3 epochs with a dataset of ~1k samples took 01:31:58 on a\ng6.2xlarge\n. The instance costs\n0.9776$/h\nwhich brings us to a total cost of only\n1.4$\n.\nPython\n# free the memory again\ndel\nmodel\ndel\ntrainer\ntorch.cuda.\nempty_cache\n()\n5. Test Model and run Inference\nAfter the training is done we want to evaluate and test our model. First we will load the base model and let it generate a description for a random Amazon product. Then we will load our Q-LoRA adapted model and let it generate a description for the same product.\nFinally we can merge the adapter into the base model to make it more efficient and run inference on the same product again.\nPython\nimport\ntorch\nfrom\ntransformers\nimport\nAutoProcessor, AutoModelForVision2Seq\nadapter_path =\n\"./qwen2-7b-instruct-amazon-description\"\n# Load Model base model\nmodel = AutoModelForVision2Seq.\nfrom_pretrained\n(\nmodel_id,\ndevice_map\n=\n\"auto\"\n,\ntorch_dtype\n=torch.float16\n)\nprocessor = AutoProcessor.\nfrom_pretrained\n(model_id)\nI selected a random product from Amazon and prepared a\ngenerate_description\nfunction to generate a description for the product.\nPython\nfrom\nqwen_vl_utils\nimport\nprocess_vision_info\n# sample from amazon.com\nsample = {\n\"product_name\"\n:\n\"Hasbro Marvel Avengers-Serie Marvel Assemble Titan-Held, Iron Man, 30,5 cm Actionfigur\"\n,\n\"catergory\"\n:\n\"Toys & Games | Toy Figures & Playsets | Action Figures\"\n,\n\"image\"\n:\n\"https://m.media-amazon.com/images/I/81+7Up7IWyL._AC_SY300_SX300_.jpg\"\n}\n# prepare message\nmessages = [{\n\"role\"\n:\n\"user\"\n,\n\"content\"\n: [\n{\n\"type\"\n:\n\"image\"\n,\n\"image\"\n: sample[\n\"image\"\n],\n},\n{\n\"type\"\n:\n\"text\"\n,\n\"text\"\n: prompt.\nformat\n(\nproduct_name\n=sample[\n\"product_name\"\n],\ncategory\n=sample[\n\"catergory\"\n])},\n],\n}\n]\ndef\ngenerate_description\n(\nsample\n,\nmodel\n,\nprocessor\n):\nmessages = [\n{\n\"role\"\n:\n\"system\"\n,\n\"content\"\n: [{\n\"type\"\n:\n\"text\"\n,\n\"text\"\n: system_message}]},\n{\n\"role\"\n:\n\"user\"\n,\n\"content\"\n: [\n{\n\"type\"\n:\n\"image\"\n,\n\"image\"\n: sample[\n\"image\"\n]},\n{\n\"type\"\n:\n\"text\"\n,\n\"text\"\n: prompt.\nformat\n(\nproduct_name\n=sample[\n\"product_name\"\n],\ncategory\n=sample[\n\"catergory\"\n])},\n]},\n]\n# Preparation for inference\ntext = processor.\napply_chat_template\n(\nmessages,\ntokenize\n=\nFalse\n,\nadd_generation_prompt\n=\nTrue\n)\nimage_inputs, video_inputs =\nprocess_vision_info\n(messages)\ninputs =\nprocessor\n(\ntext\n=[text],\nimages\n=image_inputs,\nvideos\n=video_inputs,\npadding\n=\nTrue\n,\nreturn_tensors\n=\n\"pt\"\n,\n)\ninputs = inputs.\nto\n(model.device)\n# Inference: Generation of the output\ngenerated_ids = model.\ngenerate\n(**inputs,\nmax_new_tokens\n=\n256\n,\ntop_p\n=\n1.0\n,\ndo_sample\n=\nTrue\n,\ntemperature\n=\n0.8\n)\ngenerated_ids_trimmed = [out_ids[len(in_ids) :]\nfor\nin_ids, out_ids\nin\nzip(inputs.input_ids, generated_ids)]\noutput_text = processor.\nbatch_decode\n(\ngenerated_ids_trimmed,\nskip_special_tokens\n=\nTrue\n,\nclean_up_tokenization_spaces\n=\nFalse\n)\nreturn\noutput_text[\n0\n]\n# let's generate the description\nbase_description =\ngenerate_description\n(sample, model, processor)\nprint(base_description)\n# you can disable the active adapter if you want to rerun it with\n# model.disable_adapters()\nAwesome it is working! Lets load our adapter and compare if with the base model.\nPython\nmodel.\nload_adapter\n(adapter_path)\n# load the adapter and activate\nft_description =\ngenerate_description\n(sample, model, processor)\nprint(ft_description)\nLets compare them side by side using a markdown table.\nPython\nimport\npandas\nas\npd\nfrom\nIPython.display\nimport\ndisplay,\nHTML\ndef\ncompare_generations\n(\nbase_gen\n,\nft_gen\n):\n# Create a DataFrame\ndf = pd.\nDataFrame\n({\n'Base Generation'\n: [base_gen],\n'Fine-tuned Generation'\n: [ft_gen]\n})\n# Style the DataFrame\nstyled_df = df.style.\nset_properties\n(**{\n'text-align'\n:\n'left'\n,\n'white-space'\n:\n'pre-wrap'\n,\n'border'\n:\n'1px solid black'\n,\n'padding'\n:\n'10px'\n,\n'width'\n:\n'250px'\n,\n# Set width to 150px\n'overflow-wrap'\n:\n'break-word'\n# Allow words to break and wrap as needed\n})\n# Display the styled DataFrame\ndisplay\n(\nHTML\n(styled_df.\nto_html\n()))\ncompare_generations\n(base_description, ft_description)\nBase Generation\nFine-tuned Generation\nIntroducing the Hasbro Marvel Avengers Series Marvel Assemble Titan-Held Iron Man Action Figure, a 30.5 cm tall action figure that is sure to bring the excitement of the Marvel Universe to life! This highly detailed Iron Man figure is perfect for fans of all ages and makes a great addition to any toy collection. With its sleek red and gold armor, this Iron Man figure is ready to take on any challenge. The Titan-Held feature allows for a more realistic and dynamic pose, making it a must-have for any Marvel fan. Whether you're a collector or just looking for a fun toy to play with, this Iron Man action figure is the perfect choice.\nUnleash the power of Iron Man with this 30.5 cm Hasbro Marvel Avengers Titan Hero Action Figure!  This highly detailed Iron Man figure is perfect for collectors and kids alike.  Features a realistic design and articulated joints for dynamic poses.  A must-have for any Marvel fan's collection!\nNice! Even though we just had ~1k samples we can see that the fine-tuning improve the product description generation. The description is way shorter and more concise, which fits our training data.\nOptional: Merge LoRA adapter in to the original model\nWhen using QLoRA, we only train adapters and not the full model. This means when saving the model during training we only save the adapter weights and not the full model. If you want to save the full model, which makes it easier to use with Text Generation Inference you can merge the adapter weights into the model weights using the\nmerge_and_unload\nmethod and then save the model with the\nsave_pretrained\nmethod. This will save a default model, which can be used for inference.\nNote: This requires > 30GB CPU Memory.\nPython\nfrom\npeft\nimport\nPeftModel\nfrom\ntransformers\nimport\nAutoProcessor, AutoModelForVision2Seq\nadapter_path =\n\"./qwen2-7b-instruct-amazon-description\"\nbase_model_id =\n\"Qwen/Qwen2-VL-7B-Instruct\"\nmerged_path =\n\"merged\"\n# Load Model base model\nmodel = AutoModelForVision2Seq.\nfrom_pretrained\n(model_id,\nlow_cpu_mem_usage\n=\nTrue\n)\n# Path to save the merged model\n# Merge LoRA and base model and save\npeft_model = PeftModel.\nfrom_pretrained\n(model, adapter_path)\nmerged_model = peft_model.\nmerge_and_unload\n()\nmerged_model.\nsave_pretrained\n(merged_path,\nsafe_serialization\n=\nTrue\n,\nmax_shard_size\n=\n\"2GB\"\n)\nprocessor = AutoProcessor.\nfrom_pretrained\n(base_model_id)\nprocessor.\nsave_pretrained\n(merged_path)\nBonus: Use TRL example script\nTRL provides a simple example script to fine-tune multimodal models. You can find the script\nhere\n. The script can be directly run from the command line and supports all the features of the\nSFTTrainer\n.\nBash\n# Tested on 8x H100 GPUs\naccelerate\nlaunch\n--config_file\n=\nexamples/accelerate_configs/deepspeed_zero3.yaml\n\\\nexamples/scripts/sft_vlm.py\n\\\n--dataset_name\nHuggingFaceH4/llava-instruct-mix-vsft\n\\\n--model_name_or_path\nllava-hf/llava-1.5-7b-hf\n\\\n--per_device_train_batch_size\n8\n\\\n--gradient_accumulation_steps\n8\n\\\n--output_dir\nsft-llava-1.5-7b-hf\n\\\n--bf16\n\\\n--torch_dtype\nbfloat16\n\\\n--gradient_checkpointing",
  "metadata": {
    "title": "How to Fine-Tune Multimodal Models or VLMs with Hugging Face TRL",
    "description": "Learn how to fine-tune multimodal models like Llama 3.2 Vision or Qwen 2 VL to create custom image-to-text generation models.",
    "keywords": null,
    "author": "Philipp Schmid",
    "og:title": "How to Fine-Tune Multimodal Models or VLMs with Hugging Face TRL",
    "og:description": "Learn how to fine-tune multimodal models like Llama 3.2 Vision or Qwen 2 VL to create custom image-to-text generation models.",
    "og:url": "https://www.philschmid.de/fine-tune-multimodal-llms-with-trl",
    "og:image": "https://www.philschmid.de/static/blog/fine-tune-multimodal-llms-with-trl/thumbnail.jpg",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "How to Fine-Tune Multimodal Models or VLMs with Hugging Face TRL",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "How to Fine-Tune Multimodal Models or VLMs with Hugging Face TRL",
    "twitter:description": "Learn how to fine-tune multimodal models like Llama 3.2 Vision or Qwen 2 VL to create custom image-to-text generation models.",
    "twitter:image": "https://www.philschmid.de/static/blog/fine-tune-multimodal-llms-with-trl/thumbnail.jpg",
    "article:published_time": "2024-09-30"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/3c43db3184f5/code/body-000.txt",
      "chars": 321,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-001.txt",
      "chars": 109,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-002.txt",
      "chars": 346,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-003.txt",
      "chars": 449,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-004.txt",
      "chars": 1313,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-005.txt",
      "chars": 663,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-006.txt",
      "chars": 151,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-007.txt",
      "chars": 284,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-008.txt",
      "chars": 2735,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-009.txt",
      "chars": 252,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-010.txt",
      "chars": 159,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-011.txt",
      "chars": 70,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-012.txt",
      "chars": 322,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-013.txt",
      "chars": 2105,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-014.txt",
      "chars": 152,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-015.txt",
      "chars": 714,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-016.txt",
      "chars": 685,
      "language": "text"
    },
    {
      "file": "attachments/3c43db3184f5/code/body-017.txt",
      "chars": 442,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/philschmid/deep-learning-pytorch-huggingface/blob/main/training/fine-tune-multimodal-llms-with-trl.ipynb",
      "label": "View Code",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct",
      "label": "Llama-3.2-11B-Vision",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/mistralai/Pixtral-12B-2409",
      "label": "Pixtral-12B",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/Qwen/Qwen2-VL-7B",
      "label": "Qwen2-VL-7B",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/allenai/Molmo-7B-D-0924",
      "label": "Molmo-7B-D-0924",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/trl/index",
      "label": "TRL",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/transformers/index",
      "label": "Transformers",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/datasets/index",
      "label": "datasets",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/philschmid/amazon-product-descriptions-vlm",
      "label": "philschmid/amazon-product-descriptions-vlm",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/models",
      "label": "Hugging Face Hub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/join",
      "label": "Hugging Face",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/trl/sft_trainer",
      "label": "SFTTrainer",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.14314",
      "label": "QLoRA",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/4bit-transformers-bitsandbytes",
      "label": "Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2305.14314.pdf",
      "label": "qlora paper",
      "type": "a"
    },
    {
      "href": "https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms",
      "label": "blog post",
      "type": "a"
    },
    {
      "href": "https://github.com/QwenLM/Qwen2-VL/blob/main/qwen-vl-utils/src/qwen_vl_utils/vision_process.py",
      "label": "process_vision_info",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/trl/blob/main/examples/scripts/sft_vlm.py",
      "label": "here",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:54Z",
  "source_url": "https://www.philschmid.de/fine-tune-multimodal-llms-with-trl",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.14314",
      "bytes": 1065470,
      "reused": true,
      "local_path": "papers/repair/df2f754581ddf63a5d62.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.14314.pdf",
      "bytes": 1065470,
      "reused": true,
      "local_path": "papers/repair/42f75ee4f85fbba6ba8a.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 20117,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/3c43db3184f5/pages/source.html",
  "local_html": "attachments/3c43db3184f5/pages/article.html",
  "body_base_path": "attachments/3c43db3184f5/pages/",
  "body_markdown_path": "attachments/3c43db3184f5/pages/article.md"
}