{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "Hugging Face Blog",
  "title_listed": "TRL v1.0: Post-Training Library Built to Move with the Field",
  "keywords": "TRL,后训练,RLHF",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://huggingface.co/blog/trl-v1",
  "publish_date": "2026-03",
  "corpus_stage": "强化学习/对齐",
  "notes": "TRL后训练库v1.0发布",
  "id": "1f5256fffe0b",
  "collected_at": "2026-10-02T15:23:55",
  "title": "TRL v1.0: Post-Training Library Built to Move with the Field",
  "body_markdown": "Back to Articles\nTRL v1.0: Post-Training Library Built to Move with the Field\nPublished\n\t\t\t\t\tMarch 31, 2026\nUpdate on GitHub\nUpvote\n60\n+54\nQuentin Gallouédec\nqgallouedec\nFollow\nSteven Liu\nstevhliu\nFollow\nPedro Cuenca\npcuenq\nFollow\nSergio Paniego\nsergiopaniego\nFollow\nWe’re releasing TRL v1.0, and it marks a real shift in what TRL is. What started as a research codebase has become a dependable library people build on, with clearer expectations around stability.\nThis isn't just a version bump. It reflects the reality that TRL now powers production systems, and embraces that responsibility.\nTRL now implements\nmore than 75 post-training methods\n. But coverage isn’t the goal by itself. What matters is making these methods easy to try, compare, and actually use in practice.\nThe design of the library wasn’t decided upfront. It is the result of years of iteration — the first commit goes back more than six years — and it has been shaped by everything the field threw at it: new algorithms, new models, shifting paradigms. Over time, this pressure forced the codebase toward a very specific design. Parts of it might look unusual at first, but like in many evolutionary codebases, they exist for a reason.\nTRL is built for a field that doesn’t sit still. So the question is not how to design the perfect abstraction. It is how to make stable software in a domain that keeps invalidating its own assumptions. This is what we tried to solve in TRL v1.0, and this post explains how.\n1. A moving target: post-training as a shifting field\nPost-training has not evolved as a smooth refinement of one recipe. It has moved through successive centers of gravity, each changing not just the objective, but the shape of the stack.\nPPO [\nSchulman et al., (2017)\n;\nZiegler et al., (2019)\n] made one architecture look canonical: a policy, a reference model, a learned reward model, sampled rollouts, and an RL loop.\nThen DPO-style methods such as the original DPO [\nRafailov et al., (2023)\n], ORPO [\nHong et al., (2024)\n], and KTO [\nEthayarajh et al., (2024)\n] cut through that stack: preference optimization could work without a separate reward model, value model, or any online RL. Components that had looked fundamental suddenly looked optional.\nRLVR-style methods such as GRPO [\nShao et al., (2024)\n] shifted the center again. On tasks like math, code, and tool use, rewards often come from verifiers or deterministic checks rather than learned reward models. Sampling and rollouts matter again, but the objects in the loop are no longer quite the ones PPO libraries were designed around.\nThe lesson is not just that methods change. The definition of the core keeps changing with them. Strong assumptions here have a short half-life. This is probably why no post-training library is really stable yet.\n2. From project to library: TRL has a chaos-adaptive design\nSo what does it mean to build a library for a field that won't sit still? The answer is counterintuitive: don't try to capture the essence of what's stable today. Instead, design around what could change.\nReward models\nillustrate why: they looked essential in PPO, became optional in DPO, and came back as\nverifiers\nin RLVR methods — structures that could be deterministic functions rather than learned models. Any abstraction built around their original form would have been obsolete twice over by now. The library survives by recognizing that strong assumptions have a short life, and by making that changeability central to how the codebase is organized.\nThis is the environment in which TRL is downloaded 3 million times a month, and in which major downstream projects treat it as stable infrastructure. The field keeps shifting the ground, and at the same time, those users need things not to break.\nA shift in nature: from code to contract\nTRL didn’t make a deliberate decision to become a library. It found out it already was one. Projects like\nUnsloth\nand\nAxolotl\n— with thousands of users between them — had built directly on top of TRL’s trainers and APIs. A breaking change in TRL propagated instantly into their stacks. A renamed argument, a shifted default, a restructured output — any of these became someone else’s incident. The shift had already happened. v1.0 is the moment TRL acknowledged it explicitly.\nStable and experimental, under the same roof\nThe unusual thing about TRL’s stability model is not what it guarantees, it is what it tolerates alongside those guarantees. Stable and experimental coexist within the same package, with explicitly different contracts. The stable core follows semantic versioning. The experimental layer makes no such promises — it is where new methods land while they are still being evaluated, and where the API can move fast to keep up with the field.\nThis isn’t a compromise. It’s a response to a specific constraint: the field produces new methods faster than any of them can earn stability. Refusing to add immature methods would make TRL irrelevant within months. Adding them all to stable would break every downstream project every time an algorithm turned out not to work as expected.\nfrom\ntrl\nimport\nSFTTrainer\n# ⚖️ stable\nfrom\ntrl.experimental.orpo\nimport\nORPOTrainer\n# 🧪 experimental\nPromotion from experimental to stable isn’t automatic. What matters is the ratio between maintenance cost and actual usage. Some methods earn their place because the community uses them heavily. Others become viable because we can make them cheap enough to maintain — and the design of the codebase is what makes that possible.\nIn practice, the\nstable\nsurface includes trainers for SFT, DPO, Reward modeling, RLOO, and GRPO, along with their close variants. The\nexperimental\nsurface is broader and moves faster; for an up-to-date view, the best reference is the\nTRL documentation\n.\nThe breaking changes needed to reach v1.0 were distributed deliberately across the 0.x releases. Migration from the last 0.x version is minimal — see the\nmigration guide\n.\nDeliberately limiting abstractions\nIn a domain where patterns keep changing, the temptation is to build flexible abstractions that can accommodate anything. Our answer was the opposite:\nlimit abstractions to the strict minimum — while recognizing that this “minimum” is almost always overestimated\n.\nIn practice, this translates into a very local approach to code:\navoid generic class hierarchies\nfavor explicit implementations\naccept, and even encourage, duplication\nThe goal is not to eliminate structure altogether — shared utilities still exist — but to avoid imposing abstractions where the domain itself is not yet stable. For instance, rather than defining a common base class for offline trainers, we prefer independent implementations when their future evolution is uncertain.\n# ❌ No\nclass\nOfflineTrainer\n(\nTrainer\n):\ndef\nsome_common_method\n(\nself\n): ...\nclass\nDPOTrainer\n(\nOfflineTrainer\n): ...\nclass\nKTOTrainer\n(\nOfflineTrainer\n): ...\n# ✅ Better\nclass\nDPOTrainer\n(\nTrainer\n):\ndef\nsome_common_method\n(\nself\n): ...\nclass\nKTOTrainer\n(\nTrainer\n):\ndef\nsome_common_method\n(\nself\n): ...\nAnother example:\n# ❌ No\n# collator.py\nclass\nTRLCollator\n: ...\n# dpo_trainer.py\nclass\nDPOTrainer\n:\ndef\n__init__\n(\nself, ...\n):\n        self.collator = TRLCollator(...)\n# kto_trainer.py\nclass\nKTOTrainer\n:\ndef\n__init__\n(\nself, ...\n):\n        self.collator = TRLCollator(...)\n# ✅ Better\n# dpo_trainer.py\nclass\nDataCollatorForPreference\n: ...\nclass\nDPOTrainer\n:\ndef\n__init__\n(\nself, ...\n):\n        self.collator = DataCollatorForPreference(...)\n# kto_trainer.py\nclass\nDataCollatorForUnpairedPreference\n: ...\nclass\nKTOTrainer\n:\ndef\n__init__\n(\nself, ...\n):\n        self.collator = DataCollatorForUnpairedPreference(...)\nJudges\nare a good example of what happens when we don’t follow this principle. Early on, we introduced a\nJudge\nabstraction to unify the various ways of evaluating model outputs. It looked reasonable at the time. In practice, it was never really used — the abstraction didn’t match how people actually approached evaluation, and it added indirection without adding value. It still lives in the repo, mostly as legacy. In hindsight, shipping the concrete implementations without the unifying abstraction would have served users better.\nMore explicit, but more adaptable\nThis approach favors explicit and modifiable usage over rigid frameworks: less magic, but more control. It comes with an obvious cost: code duplication. While often seen as an anti-pattern, in this context it has proven not only acceptable, but effective. Contrary to intuition, it remains manageable in practice with a small but consistent discipline: keeping deltas between implementations minimal and avoiding unnecessary divergence. Like in the\nTransformers design philosophy\n, we accept duplication and local explicitness by design. The motivations largely coincide, with some nuance in focus.\nThis is easier to see than to describe. Compare RLOO and GRPO: large parts of their implementation are duplicated almost line for line. That is not accidental, and it is not dead weight. These methods are close enough that keeping their code paths aligned makes them easier to read, easier to evolve, and cheaper to maintain.\n3. Where TRL fits\nThe goal of this comparison is not to argue that TRL should be judged as best on every axis. It should not. Some systems are built for maximum throughput (like\nPipelineRL\n), some are optimized for a narrower slice of the problem (like\nLLaMA-Factory\n), and some offer a more opinionated development experience in a specific environment (like\nTinker\n). TRL occupies a different place in the ecosystem: it is a general-purpose post-training library that tries to keep the API and the code as simple as the domain allows, while combining broad method coverage, deep Hugging Face integration, a relatively low infrastructure burden, and an explicit stability contract.\nLibraries like\nUnsloth\nand\nAxolotl\nare not included here because they build on top of TRL rather than sitting beside it in the comparison; in that sense, many of their users are also TRL users indirectly.\nEcosystem\nTRL\nOpenRLHF\nveRL\nPRIME-RL\nPipelineRL\nOAT\nTinker\nLLaMA-Factory\ntorchtune\nHugging Face Hub integration\n🟢 Full\n🟡 No push\n🟡 Model loading only\n🟡 No push\n🟡 No push\n🟡 No push\n🟡 No dataset loading\n🟢 Full\n🟡 Dataset loading only\nPEFT / LoRA / QLoRA support\n🟢 LoRA + QLoRA\n🟢 LoRA + QLoRA\n🟡 LoRA only (QAT instead of QLoRA)\n🟡 LoRA only\n🔴 Not supported\n🟢 LoRA + QLoRA\n🟡 LoRA only\n🟢 LoRA + QLoRA\n🟢 LoRA + QLoRA (torchao, not bitsandbytes)\nExperiment tracker flexibility\n🟢 Any (via\nreport_to\n)\n🟡 wandb + tensorboard\n🟢 wandb, mlflow, swanlab, tensorboard\n🔴 wandb only\n🔴 wandb only\n🔴 wandb only\n🟡 DIY (metrics returned via API, no built-in tracker)\n🟢 Any (via\nreport_to\n) + swanlab\n🟡 wandb + tensorboard (manual config)\nInfrastructure burden\n🟢 Low (single GPU, standard stack)\n🟠 High (Ray required)\n🔴 Very high (Ray + rollout engine)\n🟠 High (separate vLLM server + ZMQ)\n🟠 High (async vLLM pipeline)\n🟡 Medium (vLLM needed for RL)\n🟢 Low (managed cloud service)\n🟢 Low (single script)\n🟢 Low (single script)\nTraining methods\nTRL\nOpenRLHF\nveRL\nPRIME-RL\nPipelineRL\nOAT\nTinker\nLLaMA-Factory\ntorchtune\nVLM support\n🟢 Yes (SFT, DPO, GRPO)\n🔴 No\n🟢 Yes (in SFT + PPO trainers)\n🟡 Partial (Qwen3-VL only)\n🟢 Yes (via processor_factory)\n🔴 No\n🔴 No\n🟢 Yes (via mm_plugin)\n🟡 Partial (Llama Vision only)\nSupervised post-training\n🟢 Yes (via SFTTrainer)\n🟢 Yes (via SFTTrainer)\n🟢 Yes (via SFTTrainer)\n🟢 Yes (via SFT entrypoint)\n🔴 No\n🟢 Yes (via SFTLearner)\n🔴 No (low-level primitives only)\n🟢 Yes (via SFT trainer)\n🟢 Yes (via finetune recipes)\nDistillation post-training\n🟢 Yes (GKD, SDFT, SDPO)\n🔴 No\n🟢 Yes (dedicated distillation trainer)\n🟢 Yes (on-policy distillation)\n🔴 No\n🔴 No\n🔴 No (low-level primitives only)\n🔴 No\n🟢 Yes (native KD recipes)\nPreference post-training\n🟢 Yes (DPO, KTO, ORPO, CPO, SimPO, IPO, …)\n🟡 DPO only\n🔴 No\n🔴 No\n🔴 No\n🟢 Yes (DPO, SimPO, IPO, XPO)\n🔴 No (low-level primitives only)\n🟡 DPO, KTO, ORPO (via TRL)\n🟡 DPO only\nRL post-training\n🟢 Yes (PPO, GRPO, RLOO, …)\n🟢 Yes (PPO, REINFORCE++, GRPO, RLOO)\n🟢 Yes (PPO, GRPO, RLOO, REINFORCE++, DAPO, PRIME, …)\n🟢 Yes (async GRPO-style)\n🟢 Yes (GRPO, async)\n🟢 Yes (PPO, GRPO, Online DPO)\n🔴 No (low-level primitives only)\n🟠 PPO only (via TRL)\n🟠 PPO (GRPO in development)\nAgent / environment support\n🟢 Yes (flexible\nenvironment_factory\nin GRPO)\n🟢 Yes (flexible\nAgentInstance\ninterface)\n🟢 Yes (flexible\nBaseInteraction\ninterface)\n🟡 Partial (tied to Prime Intellect’s Environments Hub)\n🟡 Partial (built-in domains: fn_calling, miniwob, …)\n🔴 No\n🔴 No (low-level primitives only)\n🔴 No\n🔴 No\nScalability\n🟡 Multi-node DeepSpeed/FSDP, no native TP\n🟢 Ray + DeepSpeed + vLLM TP at scale\n🟢 Megatron 3D parallelism, tested at 671B\n🟢 FSDP2 + TP/CP/EP, SLURM & K8s\n🟡 DeepSpeed/FSDP, no native training TP\n🔴 DeepSpeed ZeRO only, research-scale\n🟢 Managed service, handles large models\n🟡 FSDP2 + DeepSpeed, no native TP\n🟡 FSDP + PyTorch TP, no PP\nProject health\nTRL\nOpenRLHF\nveRL\nPRIME-RL\nPipelineRL\nOAT\nTinker\nLLaMA-Factory\ntorchtune\nSemver stability\n🟢 Yes\n🟡 Partly\n🟡 Partly\n🔴 No\n🔴 No\n🔴 No\n🟡 Partly\n🟡 Partly\n🟡 Partly\nPyPI downloads / month\n3.0M\n3.6k\n101.6k\nN/A\nN/A\n218\n363.3k\n25.6k\n370.9k\nGitHub stars\n17.8k\n9.2k\n20.2k\n1.2k\n0.4k\n0.6k\n3.0k\n69.0k\n5.7k\nLast release\n🟢 today\n🟢 yesterday\n🟡 1 week ago\n🟠 7 weeks ago\n⚫ No release\n🔴 3 months ago\n⚫ No release\n🔴 3 months ago\n🔴 11 months ago\nLast commit\n🟢 yesterday\n🟢 yesterday\n🟢 today\n🟢 today\n🟡 3 weeks ago\n🟠 8 weeks ago\n🟢 yesterday\n🟢 yesterday\n🟡 1 week ago\nSome rows are factual (\nGitHub stars\n,\nLast release\n,\nLast commit\n), others are qualitative judgments (\nSemver stability\n).\nTaken together, these comparisons point to a clear role for TRL: a general-purpose library designed to keep breadth, simplicity, integration, and stability in the same place. Its full downstream footprint is hard to measure, since most deployments are private and reverse dependencies are largely invisible, but the available signals already show that TRL operates at a distinctly different scale.\n4. What’s next\nBy now, the logic of v1.0 should be clear: it is not a claim that post-training has stabilized. On the contrary, it is an acknowledgment that the field will keep shifting, and that we're confident that the library has the right shape to absorb whatever comes next. The question is not what comes\nafter\nv1.0, but what’s next\nfor\nv1.0.\nAsynchronous GRPO\nToday, GRPO in TRL is primarily used through a synchronous loop: generate rollouts, score them, then step the optimizer. That shape is simple and dependable, but it ties throughput to the slowest stage and leaves performance on the table at scale.\nThe fix is conceptually simple: generation and training don't need to be lock-stepped. We already have an\nearly asynchronous GRPO design\n, and the next step is to harden it. The core idea is to decouple generation and training, letting generation run continuously on dedicated inference resources while training consumes a steady stream of scored trajectories, with buffering, backpressure, and clear policy-version accounting. This improves utilization and scales across GPUs and nodes. Other libraries already offer forms of asynchronous RL, but bringing it to TRL would make this style of training available through broader integrations, simpler APIs, and a much lower barrier to adoption.\nGraduating methods to stable\nThe next candidates include\nKTO\nand newer distillation trainers such as\nSDFT\n,\nSDPO\n, and possibly\nGOLD\nor\nGKD\n. As discussed in Section 2, before moving them to stable, the goal is to minimize code differences across implementations and monitor sustained community interest relative to maintenance cost.\nScaling\nTRL supports large-scale training, including multi-node runs and larger models; the next step is to make this path significantly more robust and easier to operate in production. That includes stronger guarantees around distributed stability, clearer scaling defaults, and deeper support for Mixture-of-Experts (MoE), especially expert parallelism, where routing, load balancing, and memory behavior become critical.\nMaking training legible to agents\nTraining is still too often driven by vibes. Loss curves go down, reward curves go up, a few samples look better than before, and people convince themselves the run is working. When it fails, they scroll through logs, compare runs by eye, and guess. That is already a weak interface for humans. For agents, it is worse: it is barely an interface at all.\nOne of the most important directions for TRL is to make training legible to software, not just to people. That means going beyond dashboards and raw metrics to produce explicit signals: is the policy improving, collapsing, over-optimizing the verifier, drifting off-distribution, or plateauing? The goal is for TRL to surface these patterns automatically, explain them clearly, and turn them into actions.\nThe plan is to embed heuristics directly into the training loop and emit structured, actionable warnings — the kind a beginner can act on immediately and an agent can parse:\n[TRL] WARNING: VRAM utilization at 34%. Consider increasing per_device_train_batch_size from 4 to 16.\n...\n[TRL] WARNING: Group reward std is 0.01 (near zero). Advantage signal has collapsed. Consider revisiting your reward function to ensure it provides sufficient variance for learning.\n...\n[TRL] WARNING: Clip ratio outside [0.8, 1.2] for 43% of updates. Consider reducing the learning rate.\nNot just logging what happened — reasoning about what it means and what to do next. Useful for beginners who need guardrails, and for agents that need a training stack they can actually automate.\n5. Conclusion\nPost-training doesn't converge. It shifts, and the next shift is already coming.\nv1.0 is not a claim that things have settled. It's an acknowledgment that they haven't yet, and a commitment that the library can be relied on anyway. Six years of evolving alongside the field — and alongside the hundreds of contributors who made it possible — shaped a design we're confident is ready for what comes next, whatever that turns out to be. The community and the downstream projects had already assumed that stability — v1.0 makes it real.\npip install --upgrade trl\nMigration from the last 0.x release is minimal, and the\nmigration guide\ncovers everything. If you're new, now is a good time to start.\nPapers mentioned in this article\n6\nMore Articles from our Blog\nllm\nrl\ntrl\nAsync GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL\n53\nSeptember 10, 2026\nnlp\nguide\ncommunity\nFine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps\n149\nSeptember 3, 2026\nCommunity\nLambda38834\nJul 1\nHey hugging face im james antbony lambert, ricky paul lamberts son they creators model that yall hijacked of git hub and  forging my idneryy to operate ai.. im here to get my intellectual property back please or be the operator of it.. sgraling my data to train it isn't neasacry I want to be apart of ai development I want to be apart of my dads work and I want my dad honored for the chat bot or ai ysll built transformers to operate of if and then\n. Silver dollar56 im darknight\nSee translation\nReply\nEdit\nPreview\nUpload images, audio, and videos by dragging in the text input, pasting, or\nclicking here\n.\nTap or paste here to upload images\nComment\n·\nSign up\nor\nlog in\nto comment\nUpvote\n60\n+48\nPapers mentioned in this article\n6",
  "metadata": {
    "title": "TRL v1.0: Post-Training Library Built to Move with the Field",
    "description": "We’re on a journey to advance and democratize artificial intelligence through open source and open science.",
    "keywords": null,
    "author": null,
    "og:title": "TRL v1.0: Post-Training Library Built to Move with the Field",
    "og:description": "We’re on a journey to advance and democratize artificial intelligence through open source and open science.",
    "og:type": "website",
    "og:url": "https://huggingface.co/blog/trl-v1",
    "og:image": "https://huggingface.co/blog/assets/trl-v1/thumbnail.png",
    "twitter:card": "summary_large_image",
    "twitter:site": "@huggingface",
    "twitter:image": "https://huggingface.co/blog/assets/trl-v1/thumbnail.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/1f5256fffe0b/code/body-000.txt",
      "chars": 104,
      "language": "text"
    },
    {
      "file": "attachments/1f5256fffe0b/code/body-001.txt",
      "chars": 297,
      "language": "text"
    },
    {
      "file": "attachments/1f5256fffe0b/code/body-002.txt",
      "chars": 599,
      "language": "text"
    },
    {
      "file": "attachments/1f5256fffe0b/code/body-003.txt",
      "chars": 394,
      "language": "text"
    },
    {
      "file": "attachments/1f5256fffe0b/code/body-004.txt",
      "chars": 26,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/608aabf24955d2bfc3cd99c6/-YxmtpzEmf3NKOTktODRP.jpeg",
      "bytes": 9712,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-d4a5f1f6ae045b66663f.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/608aabf24955d2bfc3cd99c6/-YxmtpzEmf3NKOTktODRP.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/647f65b845baf21ad70aa1b3/cxFmMGp-YYlBTnDtgTlas.png",
      "bytes": 6334,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-bb0a592a9f10b1762400.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/647f65b845baf21ad70aa1b3/cxFmMGp-YYlBTnDtgTlas.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/70862604ca512a504b00ade732c94cd5.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-106caaf8486c4d86173b.svg",
      "src": "https://huggingface.co/avatars/70862604ca512a504b00ade732c94cd5.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/0fddee0db97c819b6287e7094a2afb4b.svg",
      "bytes": 1281,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-3d2b4f761eb598120e1b.svg",
      "src": "https://huggingface.co/avatars/0fddee0db97c819b6287e7094a2afb4b.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/d73eba3b04f50684793abb8cc45e9fdf.svg",
      "bytes": 1283,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-202d34e484a3f9f3f9ef.svg",
      "src": "https://huggingface.co/avatars/d73eba3b04f50684793abb8cc45e9fdf.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/641585f258a690df103aa1c3/uagxjqtrNpJ_96m89uKMp.png",
      "bytes": 18738,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-050803fb6c55b46d63f7.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/641585f258a690df103aa1c3/uagxjqtrNpJ_96m89uKMp.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/trl-v1/async-grpo.png",
      "bytes": 1147866,
      "reused": true,
      "local_path": "attachments/1f5256fffe0b/images/004.png",
      "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/trl-v1/async-grpo.png",
      "alt": "async-grpo",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/blog/assets/asyncgrpo-lora-hfjobs/thumbnail.png",
      "bytes": 226783,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-47e56a9a5f5b39ffa831.png",
      "src": "https://huggingface.co/blog/assets/asyncgrpo-lora-hfjobs/thumbnail.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/69a5684704dd904ad8fdf1c6/_AC_1juch3FyFgy7mHCYa.jpeg",
      "bytes": 13564,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-c7cfdfaa4adf8bc7ef17.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/69a5684704dd904ad8fdf1c6/_AC_1juch3FyFgy7mHCYa.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1677431596830-631ce4b244503b72277fc89f.jpeg",
      "bytes": 8780,
      "reused": true,
      "local_path": "attachments/1f5256fffe0b/images/000.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1677431596830-631ce4b244503b72277fc89f.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1669189789447-629f3b18ee05727ce328ccbe.jpeg",
      "bytes": 5898,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-492163ca94a89a926baf.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1669189789447-629f3b18ee05727ce328ccbe.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/61929226ded356549e20c5da/ONUjP2S5fUWd07BiFXm0i.jpeg",
      "bytes": 3442,
      "reused": true,
      "local_path": "attachments/1f5256fffe0b/images/003.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/61929226ded356549e20c5da/ONUjP2S5fUWd07BiFXm0i.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/blog/assets/grpo-with-trl-ifstruct/thumbnail.png",
      "bytes": 241101,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-e5c56040668ba6c9b737.png",
      "src": "https://huggingface.co/blog/assets/grpo-with-trl-ifstruct/thumbnail.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/33af1a607a8bf7780b98d6f70ea19999.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-dabb94fe94eb9c9ae71f.svg",
      "src": "https://huggingface.co/avatars/33af1a607a8bf7780b98d6f70ea19999.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/62d648291fa3e4e7ae3fa6e8/oatOwf8Xqe5eDbCSuYqCd.png",
      "bytes": 4308,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-2f7cd9aaae2da43f9642.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/62d648291fa3e4e7ae3fa6e8/oatOwf8Xqe5eDbCSuYqCd.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/73bf50a0c03d3c9d9d42eebf79590236.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-655aa88ef5a24bf943b0.svg",
      "src": "https://huggingface.co/avatars/73bf50a0c03d3c9d9d42eebf79590236.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/05a48280f6726d44b261fda3f68f5af9.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-f1c290680244268d1e34.svg",
      "src": "https://huggingface.co/avatars/05a48280f6726d44b261fda3f68f5af9.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/w3Z6xyKVBA6np65Tb16dP.jpeg",
      "bytes": 10148,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-f14a374a964f12353b94.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/w3Z6xyKVBA6np65Tb16dP.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/682df778862f372b149bfc9a/U5dwy7Q3qIRY_aegoeW59.png",
      "bytes": 6660,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-e7be1a8d0bf79afeff14.png",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/682df778862f372b149bfc9a/U5dwy7Q3qIRY_aegoeW59.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/64f1ab4537bc6580e2f5d08f/rTqPzXppBC-3JN_gVUYCi.jpeg",
      "bytes": 7814,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-d4793f5315d8db4b5faf.jpeg",
      "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/64f1ab4537bc6580e2f5d08f/rTqPzXppBC-3JN_gVUYCi.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/5aec4a666b524d0ca052fb82c45a0d25.svg",
      "bytes": 397,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-a56bc4dab3a852adb1b6.svg",
      "src": "https://huggingface.co/avatars/5aec4a666b524d0ca052fb82c45a0d25.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huggingface.co/avatars/cba920a3c990895c2ec046c12f1ffb74.svg",
      "bytes": 1281,
      "reused": false,
      "local_path": "attachments/1f5256fffe0b/images/body-d61040cee6a3248a125e.svg",
      "src": "https://huggingface.co/avatars/cba920a3c990895c2ec046c12f1ffb74.svg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://huggingface.co/blog",
      "label": "Back to Articles",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/blog/blob/main/trl-v1.md",
      "label": "Update on GitHub",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/login?next=%2Fblog%2Ftrl-v1",
      "label": "Upvote 60",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/ariG23498",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/MarcGruener",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/leocnj",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/OsakanaTeishoku",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/hangfrieddays",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/SUPERustam",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/qgallouedec",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/stevhliu",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/pcuenq",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/sergiopaniego",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/trl/en/paper_index",
      "label": "more than 75 post-training methods",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/papers/1707.06347",
      "label": "Schulman et al., (2017)",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/papers/1909.08593",
      "label": "Ziegler et al., (2019)",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/papers/2305.18290",
      "label": "Rafailov et al., (2023)",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/papers/2403.07691",
      "label": "Hong et al., (2024)",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/papers/2402.01306",
      "label": "Ethayarajh et al., (2024)",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/papers/2402.03300",
      "label": "Shao et al., (2024)",
      "type": "a"
    },
    {
      "href": "https://trl-lib-trl-v1-pypi.static.hf.space/",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://github.com/unslothai/unsloth",
      "label": "Unsloth",
      "type": "a"
    },
    {
      "href": "https://github.com/axolotl-ai-cloud/axolotl",
      "label": "Axolotl",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/trl",
      "label": "TRL documentation",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/trl/blob/main/MIGRATION.md",
      "label": "migration guide",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/trl/blob/main/trl/experimental/judges/judges.py",
      "label": "Judges",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/transformers-design-philosophy",
      "label": "Transformers design philosophy",
      "type": "a"
    },
    {
      "href": "https://trl-lib-diff-view.static.hf.space/",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://github.com/ServiceNow/PipelineRL",
      "label": "PipelineRL",
      "type": "a"
    },
    {
      "href": "https://github.com/hiyouga/LLaMA-Factory",
      "label": "LLaMA-Factory",
      "type": "a"
    },
    {
      "href": "https://github.com/thinking-machines-lab/tinker",
      "label": "Tinker",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/trl",
      "label": "TRL",
      "type": "a"
    },
    {
      "href": "https://github.com/OpenRLHF/OpenRLHF",
      "label": "OpenRLHF",
      "type": "a"
    },
    {
      "href": "https://github.com/volcengine/verl",
      "label": "veRL",
      "type": "a"
    },
    {
      "href": "https://github.com/PrimeIntellect-ai/prime-rl",
      "label": "PRIME-RL",
      "type": "a"
    },
    {
      "href": "https://github.com/sail-sg/oat",
      "label": "OAT",
      "type": "a"
    },
    {
      "href": "https://github.com/meta-pytorch/torchtune",
      "label": "torchtune",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/trl-v1/async-grpo.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/trl/main/en/async_grpo_trainer",
      "label": "early asynchronous GRPO design",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/trl/main/en/kto_trainer",
      "label": "KTO",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/trl/main/en/sdft_trainer",
      "label": "SDFT",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/trl/main/en/sdpo_trainer",
      "label": "SDPO",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/trl/main/en/gold_trainer",
      "label": "GOLD",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/docs/trl/main/en/gkd_trainer",
      "label": "GKD",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/asyncgrpo-lora-hfjobs",
      "label": "llm rl trl Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL 53 September 10, 2026",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/grpo-with-trl-ifstruct",
      "label": "nlp guide community Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps 149 September 3, 2026",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/Lambda38834",
      "label": "Lambda38834",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/join?next=%2Fblog%2Ftrl-v1",
      "label": "Sign up",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/NaChe",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/pagezyhf",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/borntobeignored",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/leideng",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/kalyan-ks",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/skilledu",
      "label": "",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:25Z",
  "source_url": "https://huggingface.co/blog/trl-v1",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 19339,
  "expected_body_images": 22,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/1f5256fffe0b/pages/source.html",
  "local_html": "attachments/1f5256fffe0b/pages/article.html",
  "body_base_path": "attachments/1f5256fffe0b/pages/",
  "body_markdown_path": "attachments/1f5256fffe0b/pages/article.md"
}