{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Philipp Schmid",
  "title_listed": "DPO Align LLMs in 2024 with TRL",
  "keywords": "DPO,对齐,TRL",
  "domain": "大模型算法",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://www.philschmid.de/dpo-align-llms-in-2024-with-trl",
  "publish_date": "2024-01",
  "corpus_stage": "强化学习/对齐",
  "notes": "用TRL做DPO对齐",
  "id": "68d624613e8e",
  "collected_at": "2026-10-02T14:29:16",
  "title": "RLHF in 2024 with DPO & Hugging Face",
  "body_markdown": "RLHF in 2024 with DPO & Hugging Face\nJanuary 23, 2024\n21\nminute read\nView Code\nThis blog post walks you through how to use DPO to improve open LLMs using Hugging Face\nTRL\n,\nTransformers\n&\ndatasets\nin 2024.\nResearch and experiments suggest that DPO should only be applied after SFT. This means we need an already fine-tuned LLM, which can be aligned with DPO. In this example we will use\ncognitivecomputations/dolphin-2.1-mistral-7b\na fine-tuned Mistral 7B with ChatML template.\nSetup development environment\nCreate and prepare the preference dataset\nAlign LLM with TRL and the DPOTrainer\nTest LLM (vibe-check)\nEvaluate open LLMs on MT-Bench\nNote: This example is designed to be an introduction to DPO and TRL. It is build for a single GPU environment to guide you through the process. For production use, you should consider using a distributed environment. It should be possible to run the example on a single GPU with at least 24GB of memory by reducing the training arguments, with batch size, max seq length and run evaluation after the training.\n1. Setup development environment\nOur first step is to install Hugging Face Libraries and Pytorch, including trl, transformers and datasets. If you haven't heard of trl yet, don't worry. It is a new library on top of transformers and datasets, which makes it easier to fine-tune, rlhf, align open LLMs.\nPython\n# Install Pytorch & other libraries\n!\npip install\n\"torch==2.1.2\"\ntensorboard pillow\n# Install Hugging Face libraries\n!\npip install\n--\nupgrade \\\n\"transformers[sentencepiece]==4.37.2\"\n\\\n\"datasets==2.16.1\"\n\\\n\"accelerate==0.26.1\"\n\\\n\"evaluate==0.4.1\"\n\\\n\"bitsandbytes==0.42.0\"\n\\\n\"trl==0.7.11\"\n\\\n\"peft==0.8.2\"\nIf you are using a GPU with Ampere architecture (e.g. NVIDIA A10G or RTX 4090/3090) or newer, you can use Flash Attention.\nFlashAttention\ncan accelerate training time up to 3x.\n*Note: If your machine has less than 96GB of RAM and lots of CPU cores, reduce the number of\nMAX_JOBS\n.\nPython\nimport\ntorch;\nassert\ntorch.cuda.\nget_device_capability\n()[\n0\n] >=\n8\n,\n'Hardware not supported for Flash Attention'\n# install flash-attn\n!\npip install ninja packaging\n!\nMAX_JOBS\n=\n4\npip install flash-attn\n--\nno-build-isolation\nInstalling flash attention can take quite a bit of time (10-45 minutes).\nWe will use the\nHugging Face Hub\nas a remote model storage and automatically push our model, logs and information to the Hub during training. You must register on the\nHugging Face\nfor this. After you have an account, we will use the\nlogin\nutil from the\nhuggingface_hub\npackage to log into our account and store our token (access key) on the disk.\nPython\nfrom\nhuggingface_hub\nimport\nlogin\nlogin\n(\ntoken\n=\n\"\"\n,\n# ADD YOUR TOKEN HERE\nadd_to_git_credential\n=\nTrue\n)\n2. Create and prepare the dataset\nImproving the helpfulness or quality of LLMs through Aligning methods like DPO doesn’t come for free. Compared to traditional supervised fine-tuning (SFT) alignment methods require preference data. Preference data is crucial as it serves as a proxy against which the model's outputs are evaluated and aligned. A typical DPO dataset includes a triplet out of prompt, chosen, and rejected response. There are several ways to create such a dataset, including:\nUsing existing open-source datasets, e.g.,\nSHP\nUsing LLMs to create synthetic preferences, e.g.,\nUltrafeedback\nUsing Humans to create datasets, e.g.,\nHH\nUsing a combination of the above methods, e.g.,\nOrca DPO\nEach method has advantages and disadvantages and depends on the budget, time, and quality requirements.\nIt's important to recognize that preference datasets can inherently reflect the biases of the human/AI they are based on. To ensure broader applicability and fairness, it's crucial to incorporate a diverse range of feedback in creating these datasets.\nIn our example, we will use the\nargilla/ultrafeedback-binarized-preferences-cleaned\ndataset. The best DPO dataset represents the real-world preferences of your users or customers. If you don’t have collected preferences yet, start with your existing SFT data and use different sizes/quality LLMs to generate feedback. This method was used to create the Orca DPO dataset, where GPT-4 was used for the accepted responses and Llama 70B Chat for the rejected responses. A DPO dataset will have the following format\nJSON\n{\n\"chosen\"\n:\n\"<prompt + good response>\"\n,\n\"rejected\"\n:\n\"<prompt + worse response>\"\n}\n{\n\"chosen\"\n:\n\"<prompt + good response>\"\n,\n\"rejected\"\n:\n\"<prompt + worse response>\"\n}\n{\n\"chosen\"\n:\n\"<prompt + good response>\"\n,\n\"rejected\"\n:\n\"<prompt + worse response>\"\n}\nThe\n<pompt + good response>\nand\n<prompt + worse response>\nare representend in the\nconversational\nformat as:\nJSON\n[\n{\n\"role\"\n:\n\"system\"\n,\n\"content\"\n:\n\"You are...\"\n},\n{\n\"role\"\n:\n\"user\"\n,\n\"content\"\n:\n\"...\"\n},\n{\n\"role\"\n:\n\"assistant\"\n,\n\"content\"\n:\n\"...\"\n}\n]\nNote:\nIf the dataset includes multiple turns you need to make sure that only the last turn between chosen and rejected is different. If not, you must reduce the conversation until only the last assistant turn is different.\nThe DPOTrainer expects the inputs as triples of (prompt, chosen, rejected), where\nchosen\nand\nrejected\nare the final turn of a dialogue and the\nprompt\nis N-1 turns. Those inputs also need to be already formated with the tempalte of the model, e.g.\n<|im_start|>user\\nINSTRUCTION\\n<|im_end|>\\n<|im_start|>assistant\\n...\n.\nIn our example we are going to load our open-source dataset using the 🤗 Datasets library and then convert it into the correct format. The\nargilla/ultrafeedback-binarized-preferences-cleaned\nalready comes with the DPO format (chosen/rejected). This means we can create our triplet and templetite it usng a\ntokenizer\nand the\napply_chat_template\nmethoh. We are randomly downsampling the dataset to 11,000 training samples and 2,750 test samples\nNote: This step can be different for your use case. For example, if you might need to create the\nconversational\nformat and concate the prompt and chosen/rejected response, e.g.\nHuman:\\n ... Assistant:\\n\n.\nPython\nfrom\ntransformers\nimport\nAutoTokenizer\nfrom\ndatasets\nimport\nload_dataset\n# Load Tokenizer from the hub\nmodel_id =\n\"cognitivecomputations/dolphin-2.1-mistral-7b\"\n# replace with your model id\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(model_id)\n# Load dataset from the hub\ndataset =\nload_dataset\n(\n\"argilla/ultrafeedback-binarized-preferences-cleaned\"\n,\nsplit\n=\n\"train\"\n)\ndataset = dataset.\nshuffle\n().\nselect\n(range(\n13750\n))\ndef\nrec_extract_assistant_messages\n(\nmessages\n,\nindex\n=-\n1\n):\n\"\"\"Recursively extract the last assistant messages from the end of the conversation.\"\"\"\nif\nmessages[index][\n\"role\"\n] ==\n\"assistant\"\n:\nreturn\n[messages[index]]\nelse\n:\nreturn\nrec_extract_assistant_messages\n(messages, index-\n1\n)\n# System message used if there is no system message at the beginning of the conversation\n# Can be repelaced and modified as needed\nDEFAULT_SYSTEM_MESSAGE\n=\n\"You are Dolphin, a helpful AI assistant.\"\ndef\ncreate_triplets\n(\nexample\n,\ntokenizer\n,\ndefault_system_message\n=\nDEFAULT_SYSTEM_MESSAGE\n):\n\"\"\"Create the triplets (prompt, chosen, rejected)\"\"\"\n# Extract the N-1 turns to form the prompt\n# Prepend a system message if the first message is not a system message\nprompt_messages = example[\n\"chosen\"\n][:-\n1\n]\nif\nexample[\n\"chosen\"\n][\n0\n][\n\"role\"\n] !=\n\"system\"\n:\nprompt_messages.\ninsert\n(\n0\n, {\n\"role\"\n:\n\"system\"\n,\n\"content\"\n: default_system_message})\n# Now we extract the final assistant turn to define chosen/rejected responses\nchosen_messages =\nrec_extract_assistant_messages\n(example[\n\"chosen\"\n])\nrejected_messages =\nrec_extract_assistant_messages\n(example[\n\"rejected\"\n])\n# apply template to the messages and return the triplets\nreturn\n{\n\"prompt\"\n: tokenizer.\napply_chat_template\n(prompt_messages,\ntokenize\n=\nFalse\n),\n\"chosen\"\n: tokenizer.\napply_chat_template\n(chosen_messages,\ntokenize\n=\nFalse\n),\n\"rejected\"\n: tokenizer.\napply_chat_template\n(rejected_messages,\ntokenize\n=\nFalse\n)\n}\ndataset = dataset.\nmap\n(create_triplets,\nremove_columns\n=dataset.features,\nfn_kwargs\n={\n\"tokenizer\"\n: tokenizer})\n# split dataset into 11,000 training samples and 2,750 test samples\ndataset = dataset.\ntrain_test_split\n(\ntest_size\n=\n2750\n/\n13750\n)\n# print sample cut of\nprint(dataset[\n\"train\"\n][\n0\n][\n\"prompt\"\n][:\n50\n])\nprint(dataset[\n\"train\"\n][\n0\n][\n\"chosen\"\n][:\n50\n])\nprint(dataset[\n\"train\"\n][\n0\n][\n\"rejected\"\n][:\n50\n])\n# save datasets to disk\ndataset[\n\"train\"\n].\nto_json\n(\n\"train_dataset.json\"\n,\norient\n=\n\"records\"\n)\ndataset[\n\"test\"\n].\nto_json\n(\n\"test_dataset.json\"\n,\norient\n=\n\"records\"\n)\n3. Align LLM with TRL and the DPOTrainer\nTRL supports the DPO through a dedicated\nDPOTrainer\nfor alinging LLMs from preference data, as described in\nDirect Preference Optimization: Your Language Model is Secretly a Reward Model\n. The\nDPOTrainer\nis a subclass of the\nTrainer\nfrom the\ntransformers\nlibrary and supports all the same features, including logging, evaluation, and checkpointing.\nOne big difference to SFT is that for DPO we need an additional Reference Model, which is used for KL-Divergence to help stabilize the training. The Reference Model is normally the same model as the one we are training, but frozen. This means for DPO you need additional memory and compute resources. To keep our example efficient we will use PEFT and adatpers. We load your fine-tuned and then add a new trainable adapters. This means that we will only tune adapters and not the whole model using DPO. The origian model will be then used as reference model itself. If you want to train all parameter with DPO you need to provide a\nmodel\nand `reference_model, but this requires more memory and compute resources.\nLets start by loading our saved datasets from disk.\nPython\nfrom\ndatasets\nimport\nload_dataset\n# Load jsonl data from disk\ntrain_dataset =\nload_dataset\n(\n\"json\"\n,\ndata_files\n=\n\"train_dataset.json\"\n,\nsplit\n=\n\"train\"\n)\neval_dataset =\nload_dataset\n(\n\"json\"\n,\ndata_files\n=\n\"test_dataset.json\"\n,\nsplit\n=\n\"train\"\n)\nWe are going to train\ncognitivecomputations/dolphin-2.1-mistral-7b\n. Dolphin is a fine-tuned Mistral 7B with ChatML template support system messages. You can easily swap out the model for another model, e.g.\nMistral\nor\nMixtral\nmodels, TII\nFalcon\n, or any other LLMs by changing our\nmodel_id\nvariable.\nNote: Be aware the bigger the model the more memory it will require. In our example we will use the 7B version, which can be tuned on 24GB GPUs. If you have a smaller GPU.\nThe first step is to load the model in int-4 using\nbitsandbytes\nand then add\nPython\nimport\ntorch\nfrom\ntransformers\nimport\nAutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n# Hugging Face model id\nmodel_id =\n\"cognitivecomputations/dolphin-2.1-mistral-7b\"\n# replace with your model id\n# BitsAndBytesConfig int-4 config\nbnb_config =\nBitsAndBytesConfig\n(\nload_in_4bit\n=\nTrue\n,\nbnb_4bit_use_double_quant\n=\nTrue\n,\nbnb_4bit_quant_type\n=\n\"nf4\"\n,\nbnb_4bit_compute_dtype\n=torch.bfloat16\n)\n# Load model and tokenizer\nmodel = AutoModelForCausalLM.\nfrom_pretrained\n(\nmodel_id,\ndevice_map\n=\n\"auto\"\n,\nuse_cache\n=\nFalse\n,\nattn_implementation\n=\n\"flash_attention_2\"\n,\ntorch_dtype\n=torch.bfloat16,\nquantization_config\n=bnb_config\n)\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(model_id)\ntokenizer.pad_token = tokenizer.eos_token\ntokenizer.padding_side =\n'left'\n# to prevent errors with FA\ntokenizer.truncation_side =\n'left'\n# to prevent cutting off last generation\nCompared to the\nSFTTrainer\nthe DPOTrainer has two parameter related to dataset sizing with\nmax_prompt_length\nand\nmax_length\n. The\nmax_prompt_length\nis the maximum length of the prompt and the\nmax_length\nis the maximum length of the prompt + chosen or rejected response. Those are used for tokenization, padding and trunctation. This means if we set those wrongly our data will be potentially cut off, but if we set them too high we will waste memory and time.\nThe\nAlignment Handbook\nwhen with the\nmax_prompt_length\nof 512 and\nmax_length\nof 1024 combining it with the truncation side left (90% of data samples where in that range). Truncation side left means the beginning will be removed so we keep the important assistant response. In our example we want to cover the ~97% percentile and filter out longer samples, rather than truncating.\nPython\n#### COMMENT IN TO RECALCULATE MAX LENGTHS ####\n# from numpy import percentile\n# # lets find the p95 length of the prompt\n# prompt_length = int(percentile([len(tokenizer(x)[\"input_ids\"]) for x in train_dataset[\"prompt\"]], 95))\n# max_seq_length_chosen = int(percentile([len(tokenizer(x[\"prompt\"] + x[\"chosen\"])[\"input_ids\"]) for x in train_dataset], 95))\n# max_seq_length_rejected = int(percentile([len(tokenizer(x[\"prompt\"] + x[\"rejected\"])[\"input_ids\"]) for x in train_dataset], 95))\n# max_seq_length = max(max_seq_length_chosen, max_seq_length_rejected)\n# # filter datasets to remove samples that are too long\n# train_dataset = train_dataset.filter(lambda x: len(tokenizer(x[\"prompt\"] + x[\"chosen\"])[\"input_ids\"]) <= max_seq_length)\n# eval_dataset = eval_dataset.filter(lambda x: len(tokenizer(x[\"prompt\"] + x[\"chosen\"])[\"input_ids\"]) <= max_seq_length)\n# print(f\"len(train_dataset): {len(train_dataset)}\")\n# print(f\"len(eval_dataset): {len(eval_dataset)}\")\n# # Up the lengths to next multiple of 2, why 2? Don't know\n# prompt_length = ((prompt_length + 1) // 2) * 2\n# max_seq_length = ((max_seq_length + 1) // 2) * 2\n# print(f\"p95 prompt length: {prompt_length}\")\n# print(f\"p95 prompt + chosen length: {max_seq_length}\")\nprompt_length =\n1024\nmax_seq_length =\n1512\nNote: You could reduce the\nmax_seq_length\nto\n1512\nthis would lead to a memory reduction and then increase the batch_size.\nThe\nDPOTrainer\nsupports a native integration with\npeft\n, which makes it super easy to efficiently align LLMs using, e.g. QLoRA. We only need to create our\nLoraConfig\nand provide it to the trainer. Our\nLoraConfig\nparameters are the same as for the SFT example.\nPython\nfrom\npeft\nimport\nLoraConfig\n# LoRA config based on QLoRA paper & Sebastian Raschka experiment\npeft_config =\nLoraConfig\n(\nlora_alpha\n=\n128\n,\nlora_dropout\n=\n0.05\n,\nr\n=\n256\n,\nbias\n=\n\"none\"\n,\ntarget_modules\n=\n\"all-linear\"\n,\ntask_type\n=\n\"CAUSAL_LM\"\n,\n)\nBefore we can start our training we need to define the hyperparameters (\nTrainingArguments\n) & DPO parameters.\nBased on the\nAlignment Handbook\nwe know that we need to use a ~10-100x smaller learning rate for DPO compared to SFT. In our example we reduce the learning rate from 2e-4 (SFT) to 5e-5 (DPO) or 40x smaller.\nAnother important parameter is the\nbeta\nparameter, which is used to control the strength of the alignment. The bigger the\nbeta\nis typically something in the range of 0.1 to 0.5. A higher beta means less divergence from the initial reference model or the text generations are very similar in terms of their probability distributions. In terms of training length, we go with\n1\nepoch, which is a good starting point. There is no rule of thumb for the number of epochs, it is also related to the number of epochs used for fine-tuning.\nPython\nfrom\ntransformers\nimport\nTrainingArguments\nargs =\nTrainingArguments\n(\noutput_dir\n=\n\"doplhin-dpo\"\n,\n# directory to save and repository id\nnum_train_epochs\n=\n1\n,\n# number of training epochs\nper_device_train_batch_size\n=\n12\n,\n# batch size per device during training\nper_device_eval_batch_size\n=\n4\n,\n# batch size for evaluation\ngradient_accumulation_steps\n=\n1\n,\n# number of steps before performing a backward/update pass\ngradient_checkpointing\n=\nTrue\n,\n# use gradient checkpointing to save memory\noptim\n=\n\"adamw_torch_fused\"\n,\n# use fused adamw optimizer\nlearning_rate\n=\n5e-5\n,\n# 10x higher LR than QLoRA paper\nmax_grad_norm\n=\n0.3\n,\n# max gradient norm based on QLoRA paper\nwarmup_ratio\n=\n0.1\n,\n# warmup ratio based on QLoRA paper\nlr_scheduler_type\n=\n\"cosine\"\n,\n# use cosine learning rate scheduler\nlogging_steps\n=\n25\n,\n# log every 25 steps\nsave_steps\n=\n500\n,\n# when to save checkpoint\nsave_total_limit\n=\n2\n,\n# limit the total amount of checkpoints\nevaluation_strategy\n=\n\"steps\"\n,\n# evaluate every 1000 steps\neval_steps\n=\n700\n,\n# when to evaluate\nbf16\n=\nTrue\n,\n# use bfloat16 precision\ntf32\n=\nTrue\n,\n# use tf32 precision\npush_to_hub\n=\nFalse\n,\n# push model to hub\nreport_to\n=\n\"tensorboard\"\n,\n# report metrics to tensorboard\n)\ndpo_args = {\n\"beta\"\n:\n0.1\n,\n# The beta factor in DPO loss. Higher beta means less divergence\n\"loss_type\"\n:\n\"sigmoid\"\n# The loss type for DPO.\n}\nWe now have every building block we need to create our\nDPOTrainer\nto start then training our model.\nPython\nfrom\ntrl\nimport\nDPOTrainer\ntrainer =\nDPOTrainer\n(\nmodel,\nref_model\n=\nNone\n,\n# set to none since we use peft\npeft_config\n=peft_config,\nargs\n=args,\ntrain_dataset\n=train_dataset,\neval_dataset\n=eval_dataset,\ntokenizer\n=tokenizer,\nmax_length\n=max_seq_length,\nmax_prompt_length\n=prompt_length,\nbeta\n=dpo_args[\n\"beta\"\n],\nloss_type\n=dpo_args[\n\"loss_type\"\n],\n)\nStart training our model by calling the\ntrain()\nmethod on our\nTrainer\ninstance. This will start the training loop and train our model for 2 epochs. Since we are using a PEFT method, we will only save the adapted model weights and not the full model.\nNote: During the training we want to minimize loss and grow reward/margins metrics. Keep an eye on the reward/margins metrics, if they are not growing you might need to increase the\nbeta\nparameter or adjust the\nlearning_rate\n.\nPython\n# start training, the model will be automatically saved to the hub and the output directory\ntrainer.\ntrain\n()\n# save model at the end of training\ntrainer.\nsave_model\n()\nThe training with Flash Attention for 1 epochs with a dataset of ~10k samples took ~01:30:00 on 1x H100 GPU. You should be able to run the training on a\ng5.2xlarge\ninstance by reducing the batch_size (est. to 1) and maybe the max_seq_length (est. to 1512).\nPython\n# free the memory again\ndel\nmodel\ndel\ntrainer\ntorch.cuda.\nempty_cache\n()\nOptional: Merge LoRA adapter in to the original model\nWhen using QLoRA, we only train adapters and not the full model. This means when saving the model during training we only save the adapter weights and not the full model. If you want to save the full model, which makes it easier to use with Text Generation Inference you can merge the adapter weights into the model weights using the\nmerge_and_unload\nmethod and then save the model with the\nsave_pretrained\nmethod. This will save a default model, which can be used for inference.\nNote: You might require > 30GB CPU Memory.\nPython\n#### COMMENT IN TO MERGE PEFT AND BASE MODEL ####\n# from peft import PeftModel, PeftConfig\n# from transformers import AutoModelForCausalLM, AutoTokenizer\n# from peft import AutoPeftModelForCausalLM\n# # Load PEFT model on CPU\n# model = AutoPeftModelForCausalLM.from_pretrained(\n#     args.output_dir,\n#     torch_dtype=torch.float16,\n#     low_cpu_mem_usage=True,\n# )\n# # Merge LoRA and base model and save\n# merged_model = model.merge_and_unload()\n# merged_model.save_pretrained(args.output_dir,safe_serialization=True, max_shard_size=\"2GB\")\n4. Test LLM (vibe-check)\nAfter the training is done we want to test and evaluate or model. Evaluating Generative AI models in an open-ended way is not a trivial since 1 input can have multiple correct outputs. If you want to learn more about evaluating generative models, check out\nEvaluate LLMs and RAG a practical example using Langchain and Hugging Face\nblog post. Especially, when using RLHF techniques like DPO, it's important to \"vibe-check\" the model.\nThis means we want to manually check if the responses are more aligned with what our users or customers want. This could mean that we need to check if the responses are more helpful, more accurate, more engaging, or more informative as before. A good test here is if you have data from your SFT or previous LLMs, you can compare the outputs and see if the new model is better.\nIn our case we just check a few examples and see if the model generates helpful responses using unseen prompts.\nPython\nimport\ntorch\nfrom\npeft\nimport\nAutoPeftModelForCausalLM\nfrom\ntransformers\nimport\nAutoTokenizer, pipeline\n# Path to saved peft adapter model\n# peft_model_id = args.output_dir # or\npeft_model_id =\n\"./doplhin-dpo\"\n# Load Model with PEFT adapter\nmodel = AutoPeftModelForCausalLM.\nfrom_pretrained\n(\npeft_model_id,\ndevice_map\n=\n\"auto\"\n,\ntorch_dtype\n=torch.float16\n)\ntokenizer = AutoTokenizer.\nfrom_pretrained\n(peft_model_id)\n# load into pipeline\npipe =\npipeline\n(\n\"text-generation\"\n,\nmodel\n=model,\ntokenizer\n=tokenizer)\nWe randomely select prompts from the\nteknium/OpenHermes-2.5\ndataset and a Hugging Face special.\nPython\nprompts = [\n\"A rectangular garden has a length of 25 feet and a width of 15 feet. If you want to build a fence around the entire garden, how many feet of fencing will you need?\"\n,\n\"It's Bengay for muscle relief, a combination of methyl salicylate, menthol, and what other active ingredient commonly found in aspirin?\"\n,\n\"How can i get rid of llamas in my backyard?\"\n]\nLets iterate over the prompts and generate a response using the\ngenerate\nmethod.\nPython\nfor\nprompt\nin\nprompts:\nmessages = pipe.tokenizer.\napply_chat_template\n([{\n\"role\"\n:\n\"user\"\n,\n\"content\"\n: prompt}],\ntokenize\n=\nFalse\n)\noutputs =\npipe\n(prompt,\nmax_new_tokens\n=\n2048\n,\ndo_sample\n=\nTrue\n,\ntemperature\n=\n1.0\n,\ntop_k\n=\n50\n,\ntop_p\n=\n0.9\n,\neos_token_id\n=tokenizer.eos_token_id,\npad_token_id\n=tokenizer.pad_token_id)\nprint(\nf\n\"**Prompt**:\n\\n\n{\nprompt\n}\n\\n\n\"\n)\nprint(\nf\n\"**Generated Answer**:\n\\n\n{\noutputs[\n0\n][\n'generated_text'\n][len(prompt):].\nstrip\n()\n}\n\"\n)\nprint(\n\"===\"\n*\n10\n)\n5. Evaluate open LLMs on MT-Bench\nFor our use case we will use\nMT-Bench\n. MT-Bench is a Benchmark designed by LMSYS to test the conversation and instruction-following capabilities of large language models (LLMs). It evaluates LLMs through multi-turn conversations, focusing on their ability to engage in coherent, informative, and engaging exchanges. Since human evaluation is very expensive and time consuming, LMSYS uses GPT-4-Turbo to grade the model responses. Their paper shows as 80% agreement between strong LLM and human preferences. The\nLMSYS leaderboard\nis updated regularly (last updated February 2, 2024). MT-Bench is part of the\nFastChat Repository\n.\nMT-Bench supports two different evaluation stratgies:\nsingle-answer grading: LLM grade and give a score to model's answer directly on a scale of 10\npair-wise comparison: Compare two models and see which one is better using LLM as judge, resulting in a win-rate.\nWe are going to use the pair-wise comparison method to compare the base SFT Model with the DPO model, to see if aligning the model with DPO improved the model. Running pairwise comparison on MT-Bench includes the following steps:\nClone the FastChat Repository & install the requirements\nGenerate Responses using our SFT (original) & DPO (trained) model\nEvaluate the responses using pair-wise comparison and GPT-4-Turbo as Judge\nPlot and compare the results\nMT-Bench currenlty only support OpenAI or Anthropic as Judge, where GPT-4 is the best. If you don't have access to GPT-4 you need to use a different evaluation method. I forked the FastChat repository and added GPT-4 Turbo reference answers to keep the cost lower.\nNote: If you use this example to train different model, e.g. llama you need to make sure that your model is registered and support in FastChat. This means you need:\na registered conversation template\na moodel adapter\nused to match the model path\nregister the model adapter\nThe easiest way to do this is to fork my\nrepository\nand then add your model. In our example the base model of is\ncognitivecomputations/dolphin-2.1-mistral-7b\n, which is\nalready registered in FastChat\n.\n1. Clone the FastChat Repository & install the requirements\nLet's start by cloning the FastChat repository and installing the requirements.\nNote: Restart your notebook/kernel to clear up all GPU memory.\nPython\n# Clone main branch of FastChat\n!\ngit clone https://github.com/philschmid/FastChat.git\n# Install FastChat with model worker and llm_judge dependencies\n!\npip install -e\n\"./FastChat[model_worker,llm_judge]\"\n!\npip install matplotlib tabulate\n2. Generate Responses using our SFT (original) & DPO (trained) model\nTo Generate the responses in MT-Bench we need our directory into\nFastChat/fastchat/llm_judge\nand then run the\ngen_model_answer.py\nscript. This will generate the responses and save them into a file. We will use the default\n--max-new-token\nlength of\n1024\n, which could lead to some truncation. If you want to avoid truncation you can increase the\n--max-new-token\nlength to\n1512\nor higher.\nWe change into the\nFastChat/fastchat/llm_judge\ndirectory to run all the evaluation scripts.\nPython\nimport\nos\n%cd {os.\ngetcwd\n()}/FastChat/fastchat/llm_judge\n# should be in FastChat\nLets start with the SFT model and then the DPO model.\nNote: The answer of the models will be stored to\nFastChat/fastchat/llm_judge/data/mt_bench/model_answer\n. You might want to save them later for additional evaluation, when you have a new fine-tuned model.\nPython\n# make sure that is the correct path\nmodel_path=\n\"cognitivecomputations/dolphin-2.1-mistral-7b\"\n# model id will be used to load our conversation template https://github.com/lm-sys/FastChat/blob/1db84d0906196673db361eac50d5aa65180a0ffe/fastchat/model/model_adapter.py#L1579\nmodel_id=\n\"mistral-dolphin-sft\"\n# generate model answer\n!\npython gen_model_answer.py\n--\nmodel-id {model_id}\n--\nmodel-path {model_path}\nNote: Generating all responses can take a while, ~60 minutes or more.\nNow, we generate the responses using the DPO model.\nPython\n# Change this to where you saved the model during training, remember our current directory is FastChat/\nmodel_path=\n\"/fsx/philipp/doplhin-dpo\"\n# model id will be used to load our conversation template https://github.com/lm-sys/FastChat/blob/1db84d0906196673db361eac50d5aa65180a0ffe/fastchat/model/model_adapter.py#L1579\nmodel_id=\n\"mistral-dolphin-dpo\"\n# generate model answer\n!\npython gen_model_answer.py\n--\nmodel-id {model_id}\n--\nmodel-path {model_path}\nNote: Generating all responses can take a while, ~120 minutes or more.\n3. Evaluate the responses using pair-wise comparison and GPT-4-Turbo as Judge\nAfter we have the responses we can evaluate them using the\ngen_judgment.py\nscript. This will pairwise compare all the responses using GPT-4-Turbo and rate which response is better.\nNote: We need an\nOPENAI_API_KEY\nwith access to GPT-4 Turbo, running MT-Bench will cost ~1-2$ per model evaluation.\nPython\nopen_ai_key=\n\"sk-xxxx\"\n# replace with your openai key\n# Pairwise comparison of the two models using OpenAI's GPT-4 Turbo\n!\nOPENAI_API_KEY\n={open_ai_key} python gen_judgment.py\n--\nmodel-list\n\"mistral-dolphin-dpo\"\n\"mistral-dolphin-sft\"\n--\njudge-model\n\"gpt-4-1106-preview\"\n--\nmode\n\"pairwise-all\"\nNote: This can take ~70 minutes.\n4. Plot and compare the results\nAfter we have the results we can plot them and compare the win-rate of the SFT and DPO model.\nPython\n# Results are saved at the following location, make sure its correct\nres =\n\"./data/mt_bench/model_judgment/gpt-4-1106-preview_pair.jsonl\"\n!\npython show_result.py\n--\ninput-\nfile\n{res}\n--\nmodel-list\n\"mistral-dolphin-dpo\"\n\"mistral-dolphin-sft\"\n--\njudge-model\n\"gpt-4-1106-preview\"\n--\nmode pairwise-all\n# display plot from image file\nfrom\nPIL\nimport\nImage\ni = Image.\nopen\n(\n\"win_rate_gpt-4-1106-preview.png\"\n)\ni.\nshow\n()\nmodel\nwin\nloss\ntie\nwin_rate\nloss_rate\nwin_rate_adjusted\nmistral-dolphin-dpo\n45\n17\n98\n0.28125\n0.10625\n0.5875\nmistral-dolphin-sft\n17\n45\n98\n0.10625\n0.28125\n0.4125\nBy using DPO we were able to achieve a win-rate of 0.5875 compared to 0.4125 with the SFT model. This means by applying DPO we tuned our model to generate responses, which are more aligned with what humans/AI would prefer. This is not optimal yet, but it's a good start.\nSince the guide is only a starting point, you should consider additional evaluation methods, e.g. human evaluation or instruction-following capabilities. This means we might not have reached the full potential of the model. You should consider training for more epochs and on a larger dataset to improve the model further.\n5. Clean up the FastChat Repository\nSince we temporary cloned the FastChat repository we can now clean it up by deleting the directory.\nNote: If you want to keep your evaluation results you should save the\nmodel_answer\nand\njudgment\ndirectory.\nPython\n%cd ../../..\n# delete the cloned repository\n!\nrm -rf FastChat\nThanks for reading! If you have any questions, feel free to contact me on\nTwitter\nor\nLinkedIn\n.",
  "metadata": {
    "title": "RLHF in 2024 with DPO & Hugging Face",
    "description": "In this blog post you will learn how to align LLMs using Hugging Face TRL and RLHF through Direct Preference Optimization (DPO).",
    "keywords": null,
    "author": "Philipp Schmid",
    "og:title": "RLHF in 2024 with DPO & Hugging Face",
    "og:description": "In this blog post you will learn how to align LLMs using Hugging Face TRL and RLHF through Direct Preference Optimization (DPO).",
    "og:url": "https://www.philschmid.de/dpo-align-llms-in-2024-with-trl",
    "og:image": "https://www.philschmid.de/static/blog/dpo-align-llms-in-2024-with-trl/thumbnail.jpg",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "RLHF in 2024 with DPO & Hugging Face",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "RLHF in 2024 with DPO & Hugging Face",
    "twitter:description": "In this blog post you will learn how to align LLMs using Hugging Face TRL and RLHF through Direct Preference Optimization (DPO).",
    "twitter:image": "https://www.philschmid.de/static/blog/dpo-align-llms-in-2024-with-trl/thumbnail.jpg",
    "article:published_time": "2024-01-23"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/68d624613e8e/code/body-000.txt",
      "chars": 316,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-001.txt",
      "chars": 215,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-002.txt",
      "chars": 107,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-003.txt",
      "chars": 242,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-004.txt",
      "chars": 136,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-005.txt",
      "chars": 2428,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-006.txt",
      "chars": 231,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-007.txt",
      "chars": 874,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-008.txt",
      "chars": 1274,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-009.txt",
      "chars": 278,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-010.txt",
      "chars": 1791,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-011.txt",
      "chars": 380,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-012.txt",
      "chars": 166,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-013.txt",
      "chars": 70,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-014.txt",
      "chars": 543,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-015.txt",
      "chars": 509,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-016.txt",
      "chars": 371,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-017.txt",
      "chars": 442,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-018.txt",
      "chars": 235,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-019.txt",
      "chars": 79,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-020.txt",
      "chars": 404,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-021.txt",
      "chars": 453,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-022.txt",
      "chars": 288,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-023.txt",
      "chars": 409,
      "language": "text"
    },
    {
      "file": "attachments/68d624613e8e/code/body-024.txt",
      "chars": 62,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/dpo-align-llms-in-2024-with-trl/winrate.png",
      "bytes": 15619,
      "reused": false,
      "local_path": "attachments/68d624613e8e/images/body-0dda67b4c6cf4f2a91d5.png",
      "src": "https://www.philschmid.de/static/blog/dpo-align-llms-in-2024-with-trl/winrate.png",
      "alt": "image",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/philschmid/deep-learning-pytorch-huggingface/blob/main/training/dpo-align-llms-in-2024-with-trl.ipynb",
      "label": "View Code",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/trl/index",
      "label": "TRL",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/transformers/index",
      "label": "Transformers",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/datasets/index",
      "label": "datasets",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/cognitivecomputations/dolphin-2.1-mistral-7b",
      "label": "cognitivecomputations/dolphin-2.1-mistral-7b",
      "type": "a"
    },
    {
      "href": "https://github.com/Dao-AILab/flash-attention/tree/main",
      "label": "FlashAttention",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/models",
      "label": "Hugging Face Hub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/join",
      "label": "Hugging Face",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/stanfordnlp/SHP",
      "label": "SHP",
      "type": "a"
    },
    {
      "href": "https://www.notion.so/9de9ac96f0f94aa5aed96361a26e8bf0?pvs=21",
      "label": "Ultrafeedback",
      "type": "a"
    },
    {
      "href": "https://www.notion.so/SageMaker-bi-weekly-sync-0be2e6ba876a4599b4c0da2681dfb78f?pvs=21",
      "label": "HH",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/Intel/orca_dpo_pairs",
      "label": "Orca DPO",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned?row=0",
      "label": "argilla/ultrafeedback-binarized-preferences-cleaned",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/trl/dpo_trainer",
      "label": "DPOTrainer",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.18290",
      "label": "Direct Preference Optimization: Your Language Model is Secretly a Reward Model",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2",
      "label": "Mistral",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1",
      "label": "Mixtral",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/tiiuae/falcon-40b",
      "label": "Falcon",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/alignment-handbook",
      "label": "Alignment Handbook",
      "type": "a"
    },
    {
      "href": "https://www.philschmid.de/evaluate-llm",
      "label": "Evaluate LLMs and RAG a practical example using Langchain and Hugging Face",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/teknium/OpenHermes-2.5",
      "label": "teknium/OpenHermes-2.5",
      "type": "a"
    },
    {
      "href": "https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/README.md",
      "label": "MT-Bench",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard",
      "label": "LMSYS leaderboard",
      "type": "a"
    },
    {
      "href": "https://github.com/lm-sys/FastChat/blob/1db84d0906196673db361eac50d5aa65180a0ffe/fastchat/conversation.py",
      "label": "a registered conversation template",
      "type": "a"
    },
    {
      "href": "https://github.com/lm-sys/FastChat/blob/1db84d0906196673db361eac50d5aa65180a0ffe/fastchat/model/model_adapter.py",
      "label": "a moodel adapter",
      "type": "a"
    },
    {
      "href": "https://github.com/philschmid/FastChat",
      "label": "repository",
      "type": "a"
    },
    {
      "href": "https://twitter.com/_philschmid",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/philipp-schmid-a6a2bb196/",
      "label": "LinkedIn",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:51Z",
  "source_url": "https://www.philschmid.de/dpo-align-llms-in-2024-with-trl",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.18290",
      "bytes": 1299212,
      "reused": true,
      "local_path": "papers/repair/ba63f83d413059cb8722.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 28326,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/68d624613e8e/pages/source.html",
  "local_html": "attachments/68d624613e8e/pages/article.html",
  "body_base_path": "attachments/68d624613e8e/pages/",
  "body_markdown_path": "attachments/68d624613e8e/pages/article.md"
}