{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Maxime Labonne",
  "title_listed": "Fine-tune Llama 3 with ORPO",
  "keywords": "ORPO,偏好对齐,微调",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "部分免费/会员墙",
  "url": "https://towardsdatascience.com/fine-tune-llama-3-with-orpo-56cfab2f9ada/",
  "publish_date": "2024-04",
  "corpus_stage": "预训练/SFT",
  "notes": "",
  "id": "7bae27934b60",
  "collected_at": "2026-10-01T21:47:54",
  "title": "Fine-tune Llama 3 with ORPO | Towards Data Science",
  "body_markdown": "Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# Fine-tune Llama 3 with ORPO\nA cheaper and faster unified fine-tuning technique\n[Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\nApril 19, 20247 min read\n### _A cheaper and faster unified fine-tuning technique_\n![Image generated with DALL-E 3 by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/0DSTQkcyX56nl4qYu.png)\nImage generated with DALL-E 3 by author\nORPO is a **new exciting fine-tuning technique** that combines the traditional supervised fine-tuning and preference alignment stages into a single process. This reduces the computational resources and time required for training. Moreover, empirical results demonstrate that ORPO outperforms other alignment methods on various model sizes and benchmarks.\nIn this article, we will fine-tune the new Llama 3 8B model using ORPO with the TRL library. The code is available on [Google Colab](https://colab.research.google.com/drive/1eHNWg9gnaXErdAa8_mcvjMupbSS6rDvi?usp=sharing) and in the [LLM Course](https://github.com/mlabonne/llm-course) on GitHub.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps.\n## ⚖️ ORPO\nInstruction tuning and preference alignment are essential techniques for adapting Large Language Models (LLMs) to specific tasks. Traditionally, this involves a multi-stage process: 1/ **Supervised Fine-Tuning** (SFT) on instructions to adapt the model to the target domain, followed by 2/ **preference alignment methods** like Reinforcement Learning with Human Feedback (RLHF) or Direct Preference Optimization (DPO) to increase the likelihood of generating preferred responses over rejected ones.\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/0LlRjrJYf7rWtVxGj.png)\nImage by author\nHowever, researchers have identified a limitation in this approach. While SFT effectively adapts the model to the desired domain, it inadvertently **increases the probability of generating undesirable answers** alongside preferred ones. This is why the preference alignment stage is necessary to widen the gap between the likelihoods of preferred and rejected outputs.\n![Note how the probability of rejected responses increases during supervised fine-tuning \\(image from the ORPO paper\\).](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/0rfs4IexRUX7T6-5y.png)\nNote how the probability of rejected responses increases during supervised fine-tuning (image from the ORPO paper).\nIntroduced by [Hong and Lee (2024)](https://arxiv.org/abs/2403.07691), ORPO offers an elegant solution to this problem by combining instruction tuning and preference alignment into a single, monolithic training process. ORPO modifies the standard language modeling objective, combining the negative log-likelihood loss with an odds ratio (OR) term. This OR loss weakly penalizes rejected responses while strongly rewarding preferred ones, allowing the model to simultaneously learn the target task and align with human preferences.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/1r3V1OdKtcWJJKS6cGT8chQ.png)\nORPO has been implemented in the major fine-tuning libraries, like [TRL](https://github.com/huggingface/trl), [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl), and [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory). In the next section, we will see how to use with TRL.\n## 💻 Fine-tuning Llama 3 with ORPO\n[Llama 3](https://github.com/meta-llama/llama3/tree/main) is the latest family of LLMs developed by Meta. The models were trained on an extensive dataset of **15 trillion tokens** (compared to 2T tokens for Llama 2). Two model sizes have been released: a 70 billion parameter model and a smaller 8 billion parameter model. The 70B model has already demonstrated impressive performance, scoring 82 on the MMLU benchmark and 81.7 on the HumanEval benchmark.\nLlama 3 models also increased the context length up to 8,192 tokens (4,096 tokens for Llama 2), and potentially scale up to 32k with RoPE. Additionally, the models use a new tokenizer with a 128K-token vocabulary, reducing the number of tokens required to encode text by 15%. This vocabulary also explains the bump from 7B to 8B parameters.\n![Samples from ORPO-DPO-mix-40k \\(image by author\\).](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/0G8pGN8e3ppGj0TCa.png)\n_Samples from ORPO-DPO-mix-40k (image by author)._\nORPO requires a preference dataset, including a prompt, a chosen answer, and a rejected answer. In this example, we will use `[mlabonne/orpo-dpo-mix-40k](https://huggingface.co/datasets/mlabonne/orpo-dpo-mix-40k)`, a combination of the following high-quality DPO datasets:\n  * `[argilla/distilabel-capybara-dpo-7k-binarized](https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized)`: highly scored chosen answers >=5 (2,882 samples)\n  * `[argilla/distilabel-intel-orca-dpo-pairs](https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs)`: highly scored chosen answers >=9, not in GSM8K (2,299 samples)\n  * `[argilla/ultrafeedback-binarized-preferences-cleaned](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned)`: highly scored chosen answers >=5 (22,799 samples)\n  * `[argilla/distilabel-math-preference-dpo](https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo)`: highly scored chosen answers >=9 (2,181 samples)\n  * `[unalignment/toxic-dpo-v0.2](https://huggingface.co/datasets/unalignment/toxic-dpo-v0.2)` (541 samples)\n  * `[M4-ai/prm_dpo_pairs_cleaned](https://huggingface.co/datasets/M4-ai/prm_dpo_pairs_cleaned)` (7,958 samples)\n  * `[jondurbin/truthy-dpo-v0.1](https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1)` (1,016 samples)\n\n\nThanks to [argilla](https://huggingface.co/argilla), [unalignment](https://huggingface.co/unalignment), [M4-ai](https://huggingface.co/M4-ai), and [jondurbin](https://huggingface.co/jondurbin) for providing the source datasets.\nAs per usual, let's start by installing the required libraries:\nbash\n\n```\npip install -U transformers datasets accelerate peft trl bitsandbytes wandb\n```\n\nOnce it's installed, we can import the necessary libraries and log in to W&B (optional):\npython\n\n```\nimport gcimport osimport torchimport wandbfrom datasets import load_datasetfrom google.colab import userdatafrom peft import LoraConfig, PeftModel, prepare_model_for_kbit_trainingfrom transformers import (    AutoModelForCausalLM,    AutoTokenizer,    BitsAndBytesConfig,    TrainingArguments,    pipeline,)from trl import ORPOConfig, ORPOTrainer, setup_chat_formatwb_token = userdata.get('wandb')wandb.login(key=wb_token)\n```\n\nIf you have a recent GPU, you should also be able to use the [Flash Attention library](https://github.com/Dao-AILab/flash-attention) to replace the default eager attention implementation with a more efficient one.\npython\n\n```\nif torch.cuda.get_device_capability()[0] >= 8:    !pip install -qqq flash-attn    attn_implementation = \"flash_attention_2\"    torch_dtype = torch.bfloat16else:    attn_implementation = \"eager\"    torch_dtype = torch.float16\n```\n\nIn the following, we will load the Llama 3 8B model in 4-bit precision thanks to [bitsandbytes](https://github.com/TimDettmers/bitsandbytes). We then set the LoRA configuration using [PEFT](https://github.com/huggingface/peft) for QLoRA. I'm also using the convenient `setup_chat_format()` function to modify the model and tokenizer for [ChatML](https://huggingface.co/docs/transformers/en/chat_templating#what-template-should-i-use) support. It automatically applies this chat template, adds special tokens, and resizes the model's embedding layer to match the new vocabulary size.\nNote that you need to submit a request to access [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) and be logged in to your Hugging Face account. Alternatively, you can load ungated copies of the model, like [NousResearch/Meta-Llama-3-8B](https://huggingface.co/NousResearch/Meta-Llama-3-8B).\npython\n\n```\n# Modelbase_model = \"meta-llama/Meta-Llama-3-8B\"new_model = \"OrpoLlama-3-8B\"# QLoRA configbnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_quant_type=\"nf4\",    bnb_4bit_compute_dtype=torch_dtype,    bnb_4bit_use_double_quant=True,)# LoRA configpeft_config = LoraConfig(    r=16,    lora_alpha=32,    lora_dropout=0.05,    bias=\"none\",    task_type=\"CAUSAL_LM\",    target_modules=['up_proj', 'down_proj', 'gate_proj', 'k_proj', 'q_proj', 'v_proj', 'o_proj'])# Load tokenizertokenizer = AutoTokenizer.from_pretrained(base_model)# Load modelmodel = AutoModelForCausalLM.from_pretrained(    base_model,    quantization_config=bnb_config,    device_map=\"auto\",    attn_implementation=attn_implementation)model, tokenizer = setup_chat_format(model, tokenizer)model = prepare_model_for_kbit_training(model)\n```\n\nNow that the model is ready for training, we can take care of the dataset. We load `[mlabonne/orpo-dpo-mix-40k](https://huggingface.co/datasets/mlabonne/orpo-dpo-mix-40k)` and use the `apply_chat_template()` function to convert the \"chosen\" and \"rejected\" columns into the ChatML format. Note that I'm only using 1,000 samples and not the entire dataset, as it would take too long to run.\npython\n\n```\ndataset_name = \"mlabonne/orpo-dpo-mix-40k\"dataset = load_dataset(dataset_name, split=\"all\")dataset = dataset.shuffle(seed=42).select(range(1000))def format_chat_template(row):    row[\"chosen\"] = tokenizer.apply_chat_template(row[\"chosen\"], tokenize=False)    row[\"rejected\"] = tokenizer.apply_chat_template(row[\"rejected\"], tokenize=False)    return rowdataset = dataset.map(    format_chat_template,    num_proc= os.cpu_count(),)dataset = dataset.train_test_split(test_size=0.01)\n```\n\nFirst, we need to set a few hyperparameters:\n  * `learning_rate`: ORPO uses very low learning rates compared to traditional SFT or even DPO. This value of 8e-6 comes from the original paper, and roughly corresponds to an SFT learning rate of 1e-5 and a DPO learning rate of 5e-6. I would recommend increasing it around 1e-6 for a real fine-tune.\n  * `beta`: It is the $lambda$ parameter in the paper, with a default value of 0.1. An appendix from the original paper shows how it's been selected with an ablation study.\n  * Other parameters, like `max_length` and batch size are set to use as much VRAM as available (~20 GB in this configuration). Ideally, we would train the model for 3-5 epochs, but we'll stick to 1 here.\n\n\nFinally, we can train the model using the ORPOTrainer, which acts as a wrapper.\npython\n\n```\norpo_args = ORPOConfig(    learning_rate=8e-6,    beta=0.1,    lr_scheduler_type=\"linear\",    max_length=1024,    max_prompt_length=512,    per_device_train_batch_size=2,    per_device_eval_batch_size=2,    gradient_accumulation_steps=4,    optim=\"paged_adamw_8bit\",    num_train_epochs=1,    evaluation_strategy=\"steps\",    eval_steps=0.2,    logging_steps=1,    warmup_steps=10,    report_to=\"wandb\",    output_dir=\"./results/\",)trainer = ORPOTrainer(    model=model,    args=orpo_args,    train_dataset=dataset[\"train\"],    eval_dataset=dataset[\"test\"],    peft_config=peft_config,    tokenizer=tokenizer,)trainer.train()trainer.save_model(new_model)\n```\n\nTraining the model on these 1,000 samples took about 2 hours on an L4 GPU. Let's check the W&B plots:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/0HDi6G4O5z9rpjeEG.png)\nWhile the loss goes down, the difference between the chosen and rejects answers is not clear: the average margin and accuracy are only slightly above zero and 0.5, respectively.\nIn the original paper, the authors trained models on the `[Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf)` dataset (161k samples) for 10 epochs, which is a lot longer than our quick run. They also experimented with Llama 3 and kindly [shared their logs](https://huggingface.co/orpo-explorers/hf-llama3-8b-orpo-v0.0/tensorboard) with me (thanks [Jiwoo Hong](https://twitter.com/jiwoohong98)).\nTo end this tutorial, let's merge the QLoRA adapter with the base model and push it to the Hugging Face Hub.\npython\n\n```\n# Flush memorydel trainer, modelgc.collect()torch.cuda.empty_cache()# Reload tokenizer and modeltokenizer = AutoTokenizer.from_pretrained(base_model)model = AutoModelForCausalLM.from_pretrained(    base_model,    low_cpu_mem_usage=True,    return_dict=True,    torch_dtype=torch.float16,    device_map=\"auto\",)model, tokenizer = setup_chat_format(model, tokenizer)# Merge adapter with base modelmodel = PeftModel.from_pretrained(model, new_model)model = model.merge_and_unload()model.push_to_hub(new_model, use_temp_dir=False)tokenizer.push_to_hub(new_model, use_temp_dir=False)\n```\n\nCongrats, we finished this quick fine-tune of Llama 3: [mlabonne/OrpoLlama-3–8B](https://huggingface.co/mlabonne/OrpoLlama-3-8B). You can play with it using this [Hugging Face Space](https://huggingface.co/spaces/mlabonne/OrpoLlama-3-8B) (here's a [notebook](https://colab.research.google.com/drive/1LcVUW5wsJTO2NGmozjji5CkC--646LgC?usp=sharing) to make your own). Although the model is undertrained, as highlighted by the W&B curves, I ran some evaluations on Nous' benchmark suite using [LLM AutoEval](https://github.com/mlabonne/llm-autoeval).\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/1XLNStboeDllWwCD-XyCTXw.png)\nOur ORPO fine-tune is actually pretty decent and improves the base model's performance on every benchmark. This is encouraging and likely means that a fine-tune on the entire 40k samples would yield great results.\nThis is an exciting time for the open-source community, with more and more high-quality open-weight models being released. The gap between closed-source and open-weight models is slowly closing, and fine-tuning is an essential tool to get the best performance for your use cases.\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/16MeN5SXi4yrnNyf2O_-5zQ.png)\nImage by author\n## Conclusion\nIn this article, we introduced the ORPO algorithm and explained how it unifies the SFT and preference alignment stages into a single process. Then, we used TRL to fine-tune a Llama 3 8B model on a custom preference dataset. The final model shows encouraging results and highlights ORPO's potential as a new fine-tuning paradigm.\nI hope it was useful, and I recommend running the [Colab notebook](https://colab.research.google.com/drive/1eHNWg9gnaXErdAa8_mcvjMupbSS6rDvi?usp=sharing) to fine-tune your own Llama 3 models. In future articles, we will see how to create high-quality datasets - a point that is often overlooked. If you liked this article, please follow me on [Hugging Face](https://huggingface.co/mlabonne/) and Twitter [@maximelabonne](https://twitter.com/maximelabonne).\n## References\n  * J. Hong, N. Lee, and J. Thorne, [ORPO: Monolithic Preference Optimization without Reference Model](https://arxiv.org/abs/2403.07691). 2024.\n  * L. von Werra et al., TRL: Transformer Reinforcement Learning. GitHub, 2020. [Online]. Available: <https://github.com/huggingface/trl>\n  * Bartolome, A., Martin, G., & Vila, D. (2023). Notus. In GitHub Repository. GitHub. <https://github.com/argilla-io/notus>\n  * AI at Meta, [Introducing Meta Llama 3](https://ai.meta.com/blog/meta-llama-3/), 2024.\n\n\n* * *\n![](https://assets.insightmediagroup.io/media/1787657499317_sxy3ci.png)\nWritten By\n## [Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[See all from Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Hands On Tutorials](https://towardsdatascience.com/tag/hands-on-tutorials/)[Large Language Models](https://towardsdatascience.com/tag/large-language-models/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tune-llama-3-with-orpo-56cfab2f9ada%2F&title=Fine-tune%20Llama%203%20with%20ORPO)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tune-llama-3-with-orpo-56cfab2f9ada%2F&title=Fine-tune%20Llama%203%20with%20ORPO)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tune-llama-3-with-orpo-56cfab2f9ada%2F&text=Fine-tune%20Llama%203%20with%20ORPO)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/10/090G245GEzstRc7aU-scaled.jpg)](https://towardsdatascience.com/how-to-maximize-the-value-of-your-next-conference-and-stop-wasting-your-time-6af483ddd023/)\n## [How To Maximize the Value of Your Next Conference and Stop Wasting Your Time](https://towardsdatascience.com/how-to-maximize-the-value-of-your-next-conference-and-stop-wasting-your-time-6af483ddd023/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nIn 4 steps\n[Tyler Folkman](https://towardsdatascience.com/author/tylerfolkman/)\nOctober 17, 20194 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/0u5WkGO_WQr0Q0G6y-scaled.jpg)](https://towardsdatascience.com/lets-talk-about-graph-neural-networks-323b5e97ec1f/)\n## [Let's Talk about Graph Neural Networks](https://towardsdatascience.com/lets-talk-about-graph-neural-networks-323b5e97ec1f/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nOur weekly selection of must-read Editors' Picks and original features\n[TDS Editors](https://towardsdatascience.com/author/towardsdatascience/)\nNovember 25, 20212 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/1ABSiT821jtSch1fxxl6XOQ.png)](https://towardsdatascience.com/support-vector-machines-multinomial-example-8d9a86dde1d3/)\n## [Support Vector Machines - multinomial example](https://towardsdatascience.com/support-vector-machines-multinomial-example-8d9a86dde1d3/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nSVM's are great for classifying data you do see many binomial examples out there. Here is a good multinomial example in Matlab for you...\n[Shaun Enslin](https://towardsdatascience.com/author/shaun-enslin/)\nJuly 23, 20213 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/07/07QXezPSR8_bPQX8u.png)](https://towardsdatascience.com/what-is-going-on-with-my-gan-13a00b88519e/)\n## [What is going on with my GAN?](https://towardsdatascience.com/what-is-going-on-with-my-gan-13a00b88519e/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThe challenges, solutions, and future of GANs\n[Fabiana Clemente](https://towardsdatascience.com/author/fabiana_clemente/)\nJuly 13, 20207 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/0A49tgk0zg0vm6eCy-scaled.jpg)](https://towardsdatascience.com/introducing-scalecast-a-forecasting-library-pt-1-33b556d9b019/)\n## [Introducing Scalecast: A Forecasting Library Pt. 1](https://towardsdatascience.com/introducing-scalecast-a-forecasting-library-pt-1-33b556d9b019/)\n[](https://towardsdatascience.com/fine-tune-llama-3-with-orpo-56cfab2f9ada/)\nForecast with many models at scale\n[Michael Keith](https://towardsdatascience.com/author/mikekeith52/)\nJuly 6, 20218 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/03/1X9lfQEXhZTKqKFLOip6gvA-scaled.jpeg)](https://towardsdatascience.com/education-as-the-driver-of-human-development-in-brazil-e95f9f0124fe/)\n## [Education as the driver of human development in Brazil](https://towardsdatascience.com/education-as-the-driver-of-human-development-in-brazil-e95f9f0124fe/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nAn analysis of Brazilian census data\n[Fernando Barbalho](https://towardsdatascience.com/author/fernandobarbalho/)\nMarch 2, 20228 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0YnlPnlCROeRh82CX-scaled.jpg)](https://towardsdatascience.com/how-to-become-a-data-engineer-c0319cb226c2/)\n## [How to Become a Data Engineer](https://towardsdatascience.com/how-to-become-a-data-engineer-c0319cb226c2/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nA shortcut for beginners in 2024\n[Mike Shakhomirov](https://towardsdatascience.com/author/mshakhomirov/)\nOctober 7, 202317 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/0as0nQ6Q_KeGT-UCb-scaled.jpg)](https://towardsdatascience.com/mapping-the-pokemon-world-a-network-analysis-of-habitat-based-encounters-9b8e5fe4db0a/)\n## [Mapping the Pokemon World: A Network Analysis of Habitat-Based Encounters](https://towardsdatascience.com/mapping-the-pokemon-world-a-network-analysis-of-habitat-based-encounters-9b8e5fe4db0a/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nAn introduction to Network Analysis in Python, along with a practical example using Pokemon data from the PokeApi\n[Jacob Ingle](https://towardsdatascience.com/author/jaingle77/)\nJune 14, 202416 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/08mpimB06iWuwpFxZ.png)](https://towardsdatascience.com/claudes-computer-use-intuitively-and-exhaustively-explained-5ceca60a9da8/)\n## [Claude's Computer Use - Intuitively and Exhaustively Explained](https://towardsdatascience.com/claudes-computer-use-intuitively-and-exhaustively-explained-5ceca60a9da8/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHow Anthropic made AI that can control your computer\n[Daniel Warfield](https://towardsdatascience.com/author/danielwarfield1/)\nOctober 25, 202412 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n",
  "metadata": {
    "title": "Fine-tune Llama 3 with ORPO | Towards Data Science",
    "description": "A cheaper and faster unified fine-tuning technique",
    "keywords": null,
    "author": "Maxime Labonne",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Fine-tune Llama 3 with ORPO",
    "og:description": "A cheaper and faster unified fine-tuning technique",
    "og:url": "https://towardsdatascience.com/fine-tune-llama-3-with-orpo-56cfab2f9ada/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/0DSTQkcyX56nl4qYu.png",
    "og:image:width": "1792",
    "og:image:height": "1024",
    "og:image:type": "image/png",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Maxime Labonne",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "7 min read",
    "article:published_time": "2024-04-19T13:40:57.000Z",
    "article:modified_time": "2024-04-19T13:40:57.000Z",
    "article:author": "https://towardsdatascience.com/author/mlabonne/"
  },
  "paywall": {
    "listed_paid": true,
    "detected": false,
    "marker": "",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/7bae27934b60/code/block_000.txt",
      "chars": 76
    },
    {
      "language": "txt",
      "file": "attachments/7bae27934b60/code/block_001.txt",
      "chars": 423
    },
    {
      "language": "txt",
      "file": "attachments/7bae27934b60/code/block_002.txt",
      "chars": 225
    },
    {
      "language": "txt",
      "file": "attachments/7bae27934b60/code/block_003.txt",
      "chars": 818
    },
    {
      "language": "txt",
      "file": "attachments/7bae27934b60/code/block_004.txt",
      "chars": 481
    },
    {
      "language": "txt",
      "file": "attachments/7bae27934b60/code/block_005.txt",
      "chars": 654
    },
    {
      "language": "txt",
      "file": "attachments/7bae27934b60/code/block_006.txt",
      "chars": 579
    }
  ],
  "images": [
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/0DSTQkcyX56nl4qYu.png",
      "alt": "Image generated with DALL-E 3 by author",
      "local_path": "attachments/7bae27934b60/images/000.png",
      "bytes": 1721815,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/0LlRjrJYf7rWtVxGj.png",
      "alt": "Image by author",
      "local_path": "attachments/7bae27934b60/images/001.png",
      "bytes": 1837251,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/0rfs4IexRUX7T6-5y.png",
      "alt": "Note how the probability of rejected responses increases during supervised fine-tuning (image from the ORPO paper).",
      "local_path": "attachments/7bae27934b60/images/002.png",
      "bytes": 111261,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/0G8pGN8e3ppGj0TCa.png",
      "alt": "Samples from ORPO-DPO-mix-40k (image by author).",
      "local_path": "attachments/7bae27934b60/images/003.png",
      "bytes": 130326,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/16MeN5SXi4yrnNyf2O_-5zQ.png",
      "alt": "Image by author",
      "local_path": "attachments/7bae27934b60/images/004.png",
      "bytes": 434465,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/10/090G245GEzstRc7aU-scaled.jpg",
      "alt": "",
      "local_path": "attachments/7bae27934b60/images/005.jpg",
      "bytes": 452482,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/0u5WkGO_WQr0Q0G6y-scaled.jpg",
      "alt": "",
      "local_path": "attachments/7bae27934b60/images/006.jpg",
      "bytes": 576223,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/1ABSiT821jtSch1fxxl6XOQ.png",
      "alt": "",
      "local_path": "attachments/7bae27934b60/images/007.png",
      "bytes": 1741162,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/07/07QXezPSR8_bPQX8u.png",
      "alt": "",
      "local_path": "attachments/7bae27934b60/images/008.png",
      "bytes": 20034,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/0A49tgk0zg0vm6eCy-scaled.jpg",
      "alt": "",
      "local_path": "attachments/7bae27934b60/images/009.jpg",
      "bytes": 694861,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/03/1X9lfQEXhZTKqKFLOip6gvA-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/7bae27934b60/images/010.jpeg",
      "bytes": 289263,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0YnlPnlCROeRh82CX-scaled.jpg",
      "alt": "",
      "local_path": "attachments/7bae27934b60/images/011.jpg",
      "bytes": 253700,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/0as0nQ6Q_KeGT-UCb-scaled.jpg",
      "alt": "",
      "local_path": "attachments/7bae27934b60/images/012.jpg",
      "bytes": 357976,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/08mpimB06iWuwpFxZ.png",
      "alt": "",
      "local_path": "attachments/7bae27934b60/images/013.png",
      "bytes": 986371,
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "src": "https://www.youtube.com/c/TowardsDataScience",
      "quality": "unavailable",
      "local_path": "",
      "note": "无1080p且无720p，未交付更低清晰度"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/mlabonne/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/hands-on-tutorials/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/large-language-models/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/how-to-maximize-the-value-of-your-next-conference-and-stop-wasting-your-time-6af483ddd023/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/tylerfolkman/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/lets-talk-about-graph-neural-networks-323b5e97ec1f/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/towardsdatascience/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/support-vector-machines-multinomial-example-8d9a86dde1d3/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/shaun-enslin/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/what-is-going-on-with-my-gan-13a00b88519e/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://colab.research.google.com/drive/1eHNWg9gnaXErdAa8_mcvjMupbSS6rDvi?usp=sharing"
    },
    {
      "type": "external",
      "href": "https://github.com/mlabonne/llm-course"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-engineer"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/machine-learning"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2403.07691"
    },
    {
      "type": "external",
      "href": "https://github.com/huggingface/trl"
    },
    {
      "type": "external",
      "href": "https://github.com/OpenAccess-AI-Collective/axolotl"
    },
    {
      "type": "external",
      "href": "https://github.com/hiyouga/LLaMA-Factory"
    },
    {
      "type": "external",
      "href": "https://github.com/meta-llama/llama3/tree/main"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/argilla"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/unalignment"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/M4-ai"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/jondurbin"
    },
    {
      "type": "external",
      "href": "https://github.com/Dao-AILab/flash-attention"
    },
    {
      "type": "external",
      "href": "https://github.com/TimDettmers/bitsandbytes"
    },
    {
      "type": "external",
      "href": "https://github.com/huggingface/peft"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/docs/transformers/en/chat_templating"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T11:11:48Z",
  "source_url": "https://towardsdatascience.com/fine-tune-llama-3-with-orpo-56cfab2f9ada/",
  "delivery_status": "excluded_paid",
  "discovery_status": "excluded_paid",
  "missing_reason": "按原约定排除付费/会员全文"
}