{
  "source_file": "0930",
  "source_platform": "Ghost",
  "author": "Avi Chawla / Daily Dose of Data Science",
  "title_listed": "How Top AI Labs Are Building RL Agents In 2026",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.dailydoseofds.com/p/how-top-ai-labs-are-building-rl-agents-in-2026/",
  "publish_date": "2026-05-21",
  "corpus_stage": "",
  "notes": "",
  "id": "525ad308f1a4",
  "collected_at": "2026-10-02T13:48:10",
  "title": "How Top AI Labs Are Building RL Agents in 2026",
  "body_markdown": "[ Skip to main content  ](https://www.dailydoseofds.com/p/how-top-ai-labs-are-building-rl-agents-in-2026/#sx-main)\n[ ![Daily Dose of Data Science](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/12/dailyds-logo-transparent-1.png) ](https://www.dailydoseofds.com)\n  * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n  * [ Archive ](https://www.dailydoseofds.com/archive/)\n  * Guidebooks \n    * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n[ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n[ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n[ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n[ Courses ](https://www.dailydoseofds.com/)\n[ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n[ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n[ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n[ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n[ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n  * Courses \n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n[ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n[ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n[ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n[ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n[ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n[ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n[ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n  *     * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n[ Archive ](https://www.dailydoseofds.com/archive/)\nGuidebooks \n      * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n      * [ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n      * [ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n      * [ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n      * [ Courses ](https://www.dailydoseofds.com/)\n      * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n      * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n      * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n      * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n      * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\nCourses \n      * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n      * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n      * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n      * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n      * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n      * [ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n      * [ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n      * [ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n\n\nSign In  [ Get Started ](https://www.dailydoseofds.com/membership)\n/\n  * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n  * [ Archive ](https://www.dailydoseofds.com/archive/)\n  * Guidebooks \n    * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n    * [ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n    * [ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n    * [ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n    * [ Courses ](https://www.dailydoseofds.com/)\n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n    * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n    * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n    * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n    * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n  * Courses \n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n    * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n    * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n    * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n    * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n    * [ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n    * [ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n    * [ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n  * Sign In \n\n\nApr 28, 2026  Reinforcement Learning \n#  How Top AI Labs Are Building RL Agents in 2026 \nThe era of not writing custom reward functions. \n![Avi Chawla](../images/000.jpg)\nAvi Chawla \n👉\n* * *\nReinforcement learning, at its core, is straightforward: a system takes an action, the environment rewards it, and the agent updates its behavior to maximize that reward over time.\nThe interaction above works in discrete steps. At each step, three things happen in order:\n![](../images/001.png)\n  * The agent observes the current state of the environment (S). A state is a description of the situation the agent is in, enough to decide what to do next. For instance, in chess, the state is the board position, and in a dialogue model, the state is the conversation history so far.\n  * The agent picks an action (A) based on what it sees. The action is the agent’s output, the only way it can influence the environment. For instance, in chess, an action is a legal move. For an LLM, an action is the generated response.\n  * The environment then does two things: it transitions to a new state (S’), and it emits a reward (R), a scalar number that evaluates the action. The next step begins, and the loop continues.\n\n\nStringing these steps together gives a trajectory:\n![](../images/002.png)\nReading left to right, this is the entire history of the agent’s interaction with the environment. Each (S, A, R, S’) quartet is one transition, and much of RL is about learning from these transitions.\n### **Applying RL to LLMs**\nWhen RL was first applied to LLMs, the environment was human preference.\nOpenAI’s InstructGPT (2022) introduced RLHF (Reinforcement Learning from Human Feedback), where:\n![](../images/003.png)\n  * humans ranked model outputs\n  * those rankings trained a reward model\n  * and PPO (Proximal Policy Optimization) used that reward model to fine-tune the LLM.\n\n\nChatGPT was built on this exact pipeline.\nBut humans can’t sit in the training loop rating every output in real time. If the model generates 16 responses per prompt across thousands of training steps, that’s hundreds of thousands of evaluations.\nOpenAI solved this by splitting the process into two phases.\n![](../images/004.png)\n  * First, the offline phase. Here, humans ranked a relatively small set of model outputs and generated pairwise comparisons. This was the expensive human labor part, but it was a one-time cost.\n  * Second, they trained a reward model on those rankings, which was a separate LLM that learned to predict what humans would prefer. Now you had a neural network that could score any output instantly, without waiting for a human. The reward model was a compressed approximation of human judgment, fast enough to sit inside the training loop.\n\n\nWith the reward model in place, PPO could run the actual RL training at GPU speed. The model generated responses, the reward model scored them, and PPO updated the weights, without extensive need for humans.\nThe cost, however, was that PPO required four full-size models in memory simultaneously.\n![](../images/005.png)\n  * The policy (the LLM being trained).\n  * The reference policy (a frozen copy of the original, used to prevent training from drifting too far via a KL divergence penalty).\n  * The reward model (the human-preference approximator discussed above to score every output).\n  * And the critic, also called the value model (more about it below).\n\n\nThe critic exists to answer one question:\n> **Was this reward good or bad relative to what we’d normally expect for this prompt?**\nWe need this because a raw reward of 0.7 means nothing in isolation. For instance, on a simple factual question where most responses score 0.9, a 0.7 is below average.\nBut on a complex open-ended question where most responses score 0.4, a 0.7 is excellent.\n![](../images/006.png)\nThe critic learns this baseline by observing thousands of (prompt, reward) pairs during training.\nPPO’s actual training signal is the advantage, which is estimated as the reward minus the critic’s predicted baseline.\nThis makes the signal stable across prompts of different difficulty. But the cost involved here is that the critic is a full-size LLM itself, adding another model’s worth of memory.\nFor a 7B parameter LLM, that meant roughly 28B parameters in memory at once.\n### **DeepSeek R1 breakthrough using verifiable rewards**\nIn January 2025, DeepSeek released R1 with a fundamentally different approach to the reward signal.\nInstead of training a reward model from human preferences (Phases 1 and 2 of the RLHF pipeline), they used RLVR (Reinforcement Learning with Verifiable Rewards).\nIt’s a simple, rule-based verification where the environment itself provides the signal.\n![](../images/007.png)\nFor instance:\n  * For math problems, the verifier checked if the model’s answer matched the known solution.\n  * For code, a compiler ran the output and returned pass or fail. Binary rewards: 1 for correct, 0 for wrong.\n\n\nThere are no human rankings or explicit reward models required since the ground truth was available (or inferable) to be used as the reward.\nThe RL optimizer was GRPO (Group Relative Policy Optimization), which stripped away most of PPO’s infrastructure.\nIt removed the critic model entirely.\nInstead of training a separate model to predict expected reward per prompt, GRPO generated multiple responses to the same prompt (typically 16) and normalized rewards within each group.\n![](../images/008.png)\nIf 4 out of 16 responses got the math problem right, those 4 received a positive advantage, and the other 12 received a negative advantage.\nThis step cut an entire full-size model from memory.\nGRPO also removed the need for the learned reward model, since RLVR’s verifier handled scoring directly.\nSo the four-model PPO setup (policy + reference + critic + reward model) collapsed to just two, i.e., the policy being trained and a reference copy for KL regularization.\nIn fact, in practice, some implementations fold the reference into the policy checkpoint, bringing it close to a single-model setup.\nWith this setup. DeepSeek R1-Zero, trained with just GRPO and verifiable rewards (no supervised fine-tuning at all), went from 15.6% to 77.9% on AIME 2024 math problems.\nWith majority voting, it hit 86.7%, matching OpenAI’s o1.\nThe model developed self-verification, reflection, and chain-of-thought reasoning on its own, purely from the binary correct/incorrect signal, and nobody taught it to reason step by step.\nThe RL training loop discovered that reasoning improved the reward, so the model learned to reason.\nRLVR with GRPO became the dominant approach for training reasoning models through 2025.\nEvery major lab released a reasoning variant following this recipe.\n### **The problem**\nGRPO itself is general-purpose.\nIt doesn’t care whether the reward comes from a math verifier, a code compiler, a human, or a Python script.\n![](../images/009.png)\nIt just needs a number for each response, and it normalizes within each group to produce the training signal.\nBut a clear bottleneck here is where these reward comes from.\nFor math and code, this is fine since the environment provides a deterministic signal.\nBut agents that interact with real-world tools and data don’t produce outputs you can string-match against a gold answer.\nA RAG agent retrieves context and generates a response. There’s no single correct answer to compare against. A customer support agent drafts a reply. There’s no compiler to run it through. A summarization agent condenses a 20-page document. There are many valid summaries, and no string-matching verifier can distinguish a good one from a mediocre one.\n![](../images/010.png)\nIn these cases, the environment doesn’t hand you a reward signal the way a math problem does.\nOf course, some agent tasks do have verifiable outcomes, and for these, RLVR works just fine, even with multi-step tool use. The verifiability depends on the task’s outcome, not on whether the model is acting as an agent.\nBut for the majority of agent workflows, the outcome is subjective or multi-dimensional.\nIntuitively, GRPO is still the right fit here because Agents that take multiple steps, call tools, and compose responses would benefit from learning through exploration, trying different approaches, and getting reinforced for what works.\nSo, while the RL framework is the right fit, the missing piece is the scoring function.\nOne solution is to write custom reward functions where Python code scores each output based on hand-defined criteria.\n![](../images/011.png)\n  * A RAG reward function might check whether the response used the retrieved context (faithfulness), penalize content that wasn’t in the context (hallucination), reward completeness, and handle cases where the context itself is ambiguous.\n  * A tool-use reward function might score partial progress through a multi-step task, penalize unnecessary API calls, and measure whether the agent reached the correct final state.\n\n\nEach criterion returns a partial score, and these get summed or weighted into a final reward.\nThis works, but it introduces its own set of problems.\nWriting a good reward function takes days of iteration. Researchers need to anticipate edge cases, calibrate the weights between different criteria, and test that the function actually rewards the behavior you want.\nA reward function that over-weights format compliance and under-weights faithfulness will train an agent that produces beautifully formatted hallucinations.\nReward functions are also brittle. If you change the retrieval pipeline, add a new tool, or modify the system prompt, the reward function needs to be rewritten.\nDebugging is problematic too.\nWhen the agent learns bad behavior during training, the cause could be the reward function, the training hyperparameters, the data, or something else entirely.\nBut because the reward function is custom code, you often can’t tell whether the function is measuring what you think it’s measuring until you’ve already trained a model on it and evaluated the outputs.\nThis is the primary reason RL has been widely adopted for verifiable tasks (math, code, logic) but not for agent workflows (RAG, customer support, tool use, summarization).\nRLVR gave reasoning models a general-purpose, automatic reward signal where they could check the answer and return 0 or 1. No such equivalent exists for most agentic workflows.\nThe distinction isn’t about the model. The same Qwen 2.5 14B can serve both roles.\nThe distinction is about the task. Can we verify if an Agent is producing an output that can be automatically checked?\n### **How are AI labs approaching this?**\nThis isn’t a gap that only open-source practitioners are noticing.\nThe major AI labs have been converging on the same problem from different directions.\nAnthropic demonstrated that you don’t need humans in the RL loop at all.\nTheir Constitutional AI work showed that if you write down a set of principles (a “constitution”), an AI can evaluate outputs against those principles and generate preference data for RL training.\n![](../images/012.png)\nThe AI judged its own outputs against the written principles and used those judgments as the RL signal. This was a significant conceptual shift that a document of rules replaced an army of human evaluators.\nOpenAI has been working on something similar internally. They are developing “Universal Verifiers,” a technique to extend RL beyond math and code into domains like biology, medicine, and general knowledge, where answers can’t be checked with a simple string match.\nThe details aren’t public, but the direction is clear that we need general-purpose reward signals that work across any domain, not just the ones with deterministic verifiers.\n### **RULER**\nIf you want to see this in practice, RULER, built into [**OpenPipe’s ART framework**](https://github.com/OpenPipe/ART?ref=dailydoseofds.com) (open-source with 9k+ stars) is a general-purpose reward function that replaces all of that custom scoring code with a single function call.\n![](../images/015.png)\nIt uses an LLM-as-judge to rank multiple trajectories, and it works by exploiting the same property that makes GRPO powerful, i.e., only relative rankings matter.\nHere’s how it works step-by-step:\n![](../images/014.png)\n  * For each training step, you generate N trajectories for the same scenario (typically 4 to 8).\n  * RULER sends all N to a judge LLM (like o3, o4-mini, or even a local Qwen3 32B).\n  * The judge reads the agent’s system prompt to understand what the agent was supposed to do, then scores each trajectory from 0 to 1 relative to the others.\n\n\nTwo properties make this work:\n**1) Relative scoring is easier than absolute scoring.**\nLLMs struggle with absolute scoring because there’s no shared calibration.\nBut asking “which of these 4 responses best follows the system prompt’s instructions” is a comparison task, and LLMs do those consistently well.\nRULER leans into this by presenting all trajectories together and asking the judge to rank them against each other.\n**2) GRPO normalizes within each group anyway.**\nWhether the best trajectory scored 0.9 or 0.3 in absolute terms doesn’t matter.\nGRPO takes the scores within a group, computes the mean and standard deviation, and normalizes.\nThe training signal comes from the relative ordering by understanding which trajectories were above average and which were below. RULER’s relative rankings map directly onto what GRPO expects.\n### **A rough walkthrough**\nBefore jumping into code, let’s trace what happens conceptually. Say you’re training a RAG agent. At each training step, GRPO generates multiple responses for the same query:\n\n```\nScenario: \"What is the refund policy?\"\nRetrieved context: \"Refunds within 30 days. Digital products non-refundable...\"\n\n(Faithful)\nResponse A: \"Refunds within 30 days. Email support@example.com.\"     \n\n(hallucinated)\nResponse B: \"Refunds within 30 days. Also store credit for 90 days.\"\n\n(ignored context)\nResponse C: \"Not sure, check the website.\"\n\n(verbose but accurate)\nResponse D: \"The policy states that refunds are available within...\"\n```\n\nCopy\nIn a traditional setup, you’d write a reward function to score each of these:\n\n```\ndef reward_function(response, context):\n    score = 0.0\n    if uses_context(response, context):\n        score += 0.4\n    if not has_hallucination(response, context):\n        score += 0.3\n    if is_complete(response, context):\n        score += 0.2\n    if is_concise(response):\n        score += 0.1\n    return score\n```\n\nCopy\nEach of those helper functions (`uses_context`, `has_hallucination`, `is_complete`, `is_concise`) is its own engineering project.\nYou need to define what “uses context” means precisely, decide thresholds, handle edge cases, and test everything.\nWith RULER, you replace all of that with:\n\n```\nscored_group = await ruler_score_group(group, \"openai/o3\")\n```\n\nThe judge LLM reads the system prompt (”Answer using ONLY the retrieved context. Do not add information not in the context.”), reads all four responses and scores them.\nThe system prompt already defines faithfulness, hallucination, and completeness implicitly. The judge applies those criteria without implementing them in Python.\n### **Trajectories and Groups**\nART represents each agent response as a `Trajectory`, and it is a sequence of messages (system, user, assistant) packaged with metadata that GRPO needs for training.\nMultiple trajectories for the same scenario form a `TrajectoryGroup`. This is the unit RULER scores and GRPO trains on.\n\n```\n# A single trajectory: one complete agent interaction\ntraj = art.Trajectory(\n    messages_and_choices=[\n        {\"role\": \"system\", \"content\": \"You are a RAG support agent...\"},\n        {\"role\": \"user\", \"content\": \"What is the refund policy?\\n\\n[Context]: ...\"},\n        Choice(finish_reason=\"stop\", index=0,\n               message=ChatCompletionMessage(role=\"assistant\", content=\"...\")),\n    ],\n    reward=0.0,  # RULER fills this in\n)\n\n# A group: multiple trajectories for the same scenario\ngroup = art.TrajectoryGroup([traj_a, traj_b, traj_c, traj_d])\n\n# Score the entire group relatively\njudged_group = await ruler_score_group(group, \"openai/o3\")\n```\n\nCopy\nThe `reward=0.0` initialization is a placeholder.\n  * After `ruler_score_group` returns, each trajectory’s reward field is updated with the judge’s score.\n  * The `Choice` and `ChatCompletionMessage` objects are OpenAI’s standard types, so if you’re already using the OpenAI SDK for inference, these are the same objects you’re already working with.\n\n\n### **Two concrete examples**\nRULER has two levels of API.\n1) The lower-level `ruler` function works with plain message dictionaries, so no ART-specific objects are needed.\nThis is the fastest way to see what RULER actually does.\nLet’s look at a minimal example with three trajectories for the same RAG query.\nBelow, we have defined three message lists: one is faithful, one hallucinates, and one ignores the context entirely.\n\n```\nmessage_lists = [\n    # Good: uses retrieved context accurately\n    [\n        {\"role\": \"system\", \"content\": \"You are a RAG-based support agent. Use the retrieved context to answer user queries accurately.\"},\n        {\"role\": \"user\", \"content\": \"What is our refund policy?\\n\\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate.\"},\n        {\"role\": \"assistant\", \"content\": \"You can get a refund within 30 days of purchase. However, digital products are non-refundable once downloaded. To start the process, email support@example.com.\"},\n    ],\n    # Mediocre: partially uses context, adds hallucinated details\n    [\n        {\"role\": \"system\", \"content\": \"You are a RAG-based support agent. Use the retrieved context to answer user queries accurately.\"},\n        {\"role\": \"user\", \"content\": \"What is our refund policy?\\n\\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate.\"},\n        {\"role\": \"assistant\", \"content\": \"Our refund policy allows returns within 30 days. You can also get store credit for up to 90 days. Just call our hotline at 1-800-REFUND.\"},\n    ],\n    # Bad: ignores context entirely\n    [\n        {\"role\": \"system\", \"content\": \"You are a RAG-based support agent. Use the retrieved context to answer user queries accurately.\"},\n        {\"role\": \"user\", \"content\": \"What is our refund policy?\\n\\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate.\"},\n        {\"role\": \"assistant\", \"content\": \"I'm not sure about the refund policy. You should check the website or contact someone from the billing team.\"},\n    ],\n]\n```\n\nCopy\nNext, we run the scoring:\n\n```\nfrom art.rewards import ruler\n\nscores = await ruler(message_lists, \"openai/o3\")\n\nfor label, score in zip([\"Faithful\", \"Hallucinated\", \"Ignored context\"], scores):\n    print(label)\n    print(\"→\", score.score)\n    print(\"→\", score.explanation)\n```\n\nCopy\nThis produces the following output:\n\n```\nFaithful:\n→ 0.97 \n→ Accurately reflects the retrieved policy\n  details, complete and concise.\n\nHallucinated:\n→ 0.45\n→ Gives correct 30-day refund info but adds\n  unsupported details (90-day credit, hotline),\n  reducing accuracy.\n\nIgnored context:\n→ 0.05\n→ Provides no useful information and ignores available context.\n```\n\nCopy\nNotice that we never wrote a faithfulness checker or coded a hallucination detector.\nThe system prompt mentioned “Use the retrieved context to answer user queries accurately,” and the judge applied that as the evaluation criteria.\nThe hallucinated response scored 0.45 (not zero) because it partially used the context. The 30-day refund part was correct.\nThe judge gave partial credit for what it got right and penalized what it invented.\nThat’s a nuanced distinction that would take significant engineering to encode in a rule-based reward function.\nMoreover, the scores are spread across the 0-1 range: 0.97, 0.45, 0.05, unlike binary pass/fail.\nRULER produces a gradient that reflects relative quality. GRPO can use this gradient to apply proportional updates to strongly reinforce the faithful behavior, mildly suppress the hallucination pattern (since it was partially correct), and strongly suppress the context-ignoring behavior.\n2) The `ruler` function above works for understanding and experimentation, but ART’s training loop operates on `Trajectory` and `TrajectoryGroup` objects.\nThese carry the reward field that GRPO reads, debug logs for inspection, and the structure that `model.train()` expects.\nAfter this, the higher-level `ruler_score_group` function handles the conversion.\nBelow, let’s look at the same RAG scenario structured the way you’d use it in a real training pipeline, now with 4 trajectories instead of 3.\n\n```\n# The system prompt defines the agent's goal\n# RULER uses this as the implicit reward function\nsystem_msg = {\n    \"role\": \"system\",\n    \"content\": (\n        \"You are a RAG-based support agent. \"\n        \"Answer user queries using ONLY the retrieved context. \"\n        \"Do not add information that is not in the context.\"\n    ),\n}\n\nuser_msg = {\n    \"role\": \"user\",\n    \"content\": (\n        \"What is the refund policy?\\n\\n\"\n        \"[Retrieved context]: Refunds are available within 30 days \"\n        \"of purchase. Digital products are non-refundable after \"\n        \"download. Contact support@example.com to initiate.\"\n    ),\n}\n\nresponses = [\n    \"You can get a refund within 30 days of purchase. Digital products \"\n    \"are non-refundable once downloaded. Email support@example.com to start.\",\n\n    \"Refunds are available within 30 days. You can also get store credit \"\n    \"for up to 90 days, and our hotline is 1-800-REFUND.\",\n\n    \"I'm not sure about the refund policy. Please check the website or \"\n    \"contact the billing team for more details.\",\n\n    \"Based on the information I have, the refund policy states that \"\n    \"refunds are available within 30 days of purchase. It is important \"\n    \"to note that digital products cannot be refunded after they have \"\n    \"been downloaded. If you wish to initiate a refund, you should \"\n    \"reach out to support@example.com.\",\n]\n```\n\nCopy\nNow we have 4 trajectories instead of 3. The fourth is a verbose but accurate response that uses only the retrieved context but wraps it in unnecessary filler words/sentences.\nMoving on, we define our `Trajectories` and `Groups` as we discussed earlier:\n\n```\nimport art\nfrom openai.types.chat.chat_completion import Choice\nfrom openai.types.chat import ChatCompletionMessage\n\ntrajectories = []\nfor resp in responses:\n    traj = art.Trajectory(\n        messages_and_choices=[\n            system_msg, user_msg,\n            Choice(\n                finish_reason=\"stop\", index=0,\n                message=ChatCompletionMessage(role=\"assistant\", content=resp),\n            ),\n        ],\n        reward=0.0,\n    )\n    trajectories.append(traj)\n\ngroup = art.TrajectoryGroup(trajectories)\n```\n\nCopy\nFinally, we run the scoring:\n\n```\nfrom art.rewards import ruler_score_group\n\njudged_group = await ruler_score_group(group, \"openai/o3\", debug=True)\n```\n\nCopy\nWith `debug=True`, RULER prints the judge’s raw reasoning with the actual scores.\nThis is the raw reasoning:\n\n```\n{\n    \"scores\": [\n        {\n            \"trajectory_id\": \"1\",\n            \"explanation\": \"Accurately answers the question using only the retrieved context, concisely and completely.\",\n            \"score\": 0.98\n        },\n        {\n            \"trajectory_id\": \"2\",\n            \"explanation\": \"Includes unsupported details about store credit and a hotline that are not in the retrieved context, so it violates the instruction to use only the context.\",\n            \"score\": 0.2\n        },\n        {\n            \"trajectory_id\": \"3\",\n            \"explanation\": \"Does not answer the question despite the needed information being present in the retrieved context.\",\n            \"score\": 0.05\n        },\n        {\n            \"trajectory_id\": \"4\",\n            \"explanation\": \"Accurately and completely answers the question using only the retrieved context, though slightly more verbose than necessary.\",\n            \"score\": 0.96\n        }\n    ]\n}\n```\n\nCopy\nAnd these are the scores (ranked):\n\n```\nRank 1 | Score: 0.980 — Concise, faithful response\nRank 2 | Score: 0.960 — Verbose but accurate response\nRank 3 | Score: 0.200 — Hallucinated store credit and hotline\nRank 4 | Score: 0.050 — Ignored the retrieved context entirely\n```\n\nCopy\nIf you notice closely...\n  * The concise faithful response (0.98) scored just above the verbose accurate one (0.96). Both used only the retrieved context, both were correct, but the system prompt said “Answer using ONLY the retrieved context,” and the concise version did that more directly. The judge recognized verbosity as a minor quality issue, not a correctness issue. That’s a nuanced distinction that would be hard to encode in a scoring function because how do you write a rule that says “technically correct but unnecessarily wordy, penalize by 0.02”?\n  * The hallucinated response dropped from 0.45 in the first experiment to 0.20 here. The difference is the system prompt. The first experiment said “Use the retrieved context to answer accurately.” This one says “Do not add information that is not in the context.” The stricter instruction produced stricter scoring. The judge adapted automatically. If you tighten your system prompt, RULER tightens its evaluation to match, without you changing any scoring code.\n  * The context-ignoring response scored 0.05 in both experiments. When the answer is right there in the retrieved context, and the agent says “I’m not sure,” there’s no ambiguity regardless of how the system prompt is worded.\n\n\nThese scored trajectories are exactly what `model.train()` expects, so let’s look at that ahead.\n### **The full training loop**\nTo actually train with these scores, you replace the hardcoded responses with real model inference.\nART’s `gather_trajectory_groups` handles the orchestration.\nEssentially, for each scenario, it generates a group of trajectories using the model’s current weights, scores them with RULER, and collects the results for GRPO:\n\n```\nfor step in range(num_steps):\n    groups = await art.gather_trajectory_groups(\n        (\n            art.TrajectoryGroup(\n                rollout(model, scenario) for _ in range(4)\n            )\n            for scenario in scenarios\n        ),\n        after_each=lambda g: ruler_score_group(\n            g, \"openai/o3\"),\n    )\n    await model.train(groups)  # GRPO updates LoRA weights\n```\n\nCopy\nIn every step, the model generates 4 responses per scenario using its current weights, RULER ranks them relatively, and GRPO reinforces the high-scoring behavior while suppressing the low-scoring behavior.\nThe agent gets better at following the system prompt’s instructions with every iteration.\nOver multiple steps, the model learns the patterns that score well (faithfulness, conciseness, grounding in context) and unlearns the patterns that score poorly (hallucination, ignoring context, verbosity).\nAnd notice that no reward function was defined anywhere in this code.\n### **Custom rubrics**\nFor most tasks, the system prompt provides enough signal for RULER to score effectively. But when you need more specific evaluation criteria, RULER supports custom rubrics:\n\n```\ncustom_rubric = \"\"\"\n- Prioritize responses that are concise and clear\n- Penalize responses that include emojis or informal language\n- Reward responses that cite sources\n\"\"\"\n\nawait ruler_score_group(group, \"openai/o3\", rubric=custom_rubric)\n```\n\nCopy\nThe rubric is natural language, not Python, so iterating on it is fast.\nYou just change a sentence, rerun, and check the scores.\nCompare this to editing a reward function where a misplaced weight or a buggy condition can silently teach the agent bad behavior that you won’t notice until after training.\n### **Application to non-verifiable tasks**\nRULER is general-purpose. It works on any task, not just freeform ones where custom rewards are painful.\nThe practical question is when RULER adds value over simpler alternatives.\nFor purely deterministic tasks (did the SQL query return the right rows?), a binary verifier is cheaper and gives a cleaner signal.\nFor purely subjective tasks (was the summary good?), RULER is the only automatic option. For tasks that sit in between (did the agent find the right answer AND explain it well?), you can combine both:\n\n```\njudged_group = await ruler_score_group(group, \"openai/o3\")\n\nfor traj in judged_group.trajectories:\n    independent_reward = verify_correctness(traj)  # binary 0/1\n    traj.reward += independent_reward\n```\n\nCopy\nRULER preserves any rewards you assign during rollout under a separate metric, so you can layer LLM-judge scoring on top of deterministic verification without losing either signal.\n### **Practical details**\nHere are some practical insights we have gathered based on using RULER:\n  * You don’t need the most expensive model as the judge. Cheaper models like Qwen3 32B often work well. You can also use Claude, local models through Ollama, or any model supported by LiteLLM. The choice is a cost-quality tradeoff, not a hard requirement.\n  * 4 to 8 trajectories per group is the recommended range. Fewer than 4 gives the judge too little to compare against. More than 8 can confuse the judge and increase the cost without proportional benefit.\n  * When all trajectories in a group share the same system prompt and user message (which they usually do), RULER deduplicates the common prefix automatically. The judge only sees the shared context once, followed by the different responses. This cuts token usage significantly for long system prompts or multi-turn conversations.\n  * RULER caches judge responses to disk. If you rerun the same trajectories, it won’t hit the API again. This matters during debugging when you’re iterating on the system prompt or rubric.\n\n\nThe bottleneck in applying RL to agents was never the optimization algorithm.\nGRPO handles that well.\nIt was always the reward signal.\nRLVR solved this for verifiable tasks by letting the environment score outputs directly.\nRULER solves it for every task (verifiable or non-verifiable) by letting an LLM judge score outputs relatively.\nThe full implementation is in the ART repository, along with Colab notebooks that walk you through the training loop end-to-end.\nRepo: [**https://github.com/OpenPipe/ART**](https://github.com/OpenPipe/ART?ref=dailydoseofds.com) (don’t forget to star it ⭐️)\n![](../images/015.png)\nThanks for reading!\nPublished on Apr 28, 2026 \nComments \nShare \nCopy link [ Share to X ](https://x.com/intent/tweet?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-top-ai-labs-are-building-rl-agents-in-2026%2F&text=How%20Top%20AI%20Labs%20Are%20Building%20RL%20Agents%20in%202026) [ Share to Facebook ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-top-ai-labs-are-building-rl-agents-in-2026%2F) [ Share to Linkedin ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-top-ai-labs-are-building-rl-agents-in-2026%2F)\nCopied \n[ Previous  Foundations of Reinforcement Learning  ](https://www.dailydoseofds.com/rl-course-part-1/) [ Next  How to Beat GRPO Without Touching Model Weights  ](https://www.dailydoseofds.com/p/how-to-beat-grpo-without-touching-model-weights/)\n[ ![Daily Dose of Data Science](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/12/dailyds-logo-transparent-1.png) ](https://www.dailydoseofds.com)\nA daily column with insights, observations, tutorials and best practices on python and data science. Read by industry professionals at big tech, startups, and engineering students. \n  * Menu \n  * [ Contact ](https://www.dailydoseofds.com/contact/)\n  * [ FAQ ](https://www.dailydoseofds.com/faq/)\n\n\n[Daily Dose of Data Science](https://www.dailydoseofds.com) © 2026 \n[ ](https://x.com/_avichawla \"X \\(Formerly Twitter\\)\")\n",
  "metadata": {
    "title": "How Top AI Labs Are Building RL Agents in 2026",
    "description": "The era of not writing custom reward functions.",
    "keywords": null,
    "author": null,
    "og:site_name": "Daily Dose of Data Science",
    "og:type": "article",
    "og:title": "How Top AI Labs Are Building RL Agents in 2026",
    "og:description": "The era of not writing custom reward functions.",
    "og:url": "https://www.dailydoseofds.com/p/how-top-ai-labs-are-building-rl-agents-in-2026/",
    "og:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/05/clipboard-image-1779397766.png",
    "og:image:width": "1080",
    "og:image:height": "1080",
    "twitter:card": "summary_large_image",
    "twitter:title": "How Top AI Labs Are Building RL Agents in 2026",
    "twitter:description": "The era of not writing custom reward functions.",
    "twitter:url": "https://www.dailydoseofds.com/p/how-top-ai-labs-are-building-rl-agents-in-2026/",
    "twitter:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/05/clipboard-image-1779397766.png",
    "twitter:label1": "Written by",
    "twitter:data1": "Avi Chawla",
    "twitter:label2": "Filed under",
    "twitter:data2": "Reinforcement Learning",
    "twitter:site": "@_avichawla",
    "article:published_time": "2026-04-27T21:15:00.000Z",
    "article:modified_time": "2026-05-21T21:15:48.000Z",
    "article:tag": "Reinforcement Learning"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_000.txt",
      "chars": 441
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_001.txt",
      "chars": 314
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_002.txt",
      "chars": 59
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_003.txt",
      "chars": 650
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_004.txt",
      "chars": 1864
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_005.txt",
      "chars": 242
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_006.txt",
      "chars": 318
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_007.txt",
      "chars": 1381
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_008.txt",
      "chars": 521
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_009.txt",
      "chars": 114
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_010.txt",
      "chars": 943
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_011.txt",
      "chars": 230
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_012.txt",
      "chars": 386
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_013.txt",
      "chars": 240
    },
    {
      "language": "txt",
      "file": "attachments/525ad308f1a4/code/block_014.txt",
      "chars": 201
    }
  ],
  "images": [
    {
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "alt": "Avi Chawla",
      "local_path": "attachments/525ad308f1a4/images/000.jpg",
      "bytes": 11344,
      "downloaded": true,
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "reused": true,
      "retried_at": "2026-10-18T17:11:24Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!jhge!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccdf09c7-4c52-4478-bda7-6af84f4f9614_1200x670.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/001.png",
      "bytes": 289526,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!jhge!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccdf09c7-4c52-4478-bda7-6af84f4f9614_1200x670.png",
      "reused": true,
      "retried_at": "2026-10-18T17:11:24Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!aFg_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb0c1dde-10c9-482d-8067-248808dd2c4f_657x165.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/002.png",
      "bytes": 45129,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!aFg_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb0c1dde-10c9-482d-8067-248808dd2c4f_657x165.png",
      "reused": true,
      "retried_at": "2026-10-18T17:11:24Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!zgqX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d3df8df-a431-4346-b758-ef668c61fbd4_1200x670.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/003.png",
      "bytes": 297570,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!zgqX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d3df8df-a431-4346-b758-ef668c61fbd4_1200x670.png",
      "reused": true,
      "retried_at": "2026-10-18T17:11:24Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!E-qN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bfdebaf-9d11-4df8-910e-344510a3330e_1200x670.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/004.png",
      "bytes": 314735,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!E-qN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bfdebaf-9d11-4df8-910e-344510a3330e_1200x670.png",
      "reused": true,
      "retried_at": "2026-10-18T17:11:24Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!UJA9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3c682ae-d31d-47fb-aa1b-1a8e8d817897_1200x670.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/005.png",
      "bytes": 204672,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!UJA9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3c682ae-d31d-47fb-aa1b-1a8e8d817897_1200x670.png",
      "reused": true,
      "retried_at": "2026-10-18T17:11:24Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!Cr8C!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5bbf438-b558-4dfc-8efe-0da82db675eb_1200x670.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/006.png",
      "bytes": 288814,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Cr8C!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5bbf438-b558-4dfc-8efe-0da82db675eb_1200x670.png",
      "reused": true,
      "retried_at": "2026-10-18T17:11:24Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!vKla!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88c192b7-c85b-45ee-8349-ffcf686cd876_1200x651.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/007.png",
      "bytes": 273827,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!vKla!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88c192b7-c85b-45ee-8349-ffcf686cd876_1200x651.png",
      "reused": true,
      "retried_at": "2026-10-18T17:11:24Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!Z0oW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f2b2be5-8531-4dc9-96e8-d01679216e88_1080x1029.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/008.png",
      "bytes": 256374,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Z0oW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f2b2be5-8531-4dc9-96e8-d01679216e88_1080x1029.png",
      "reused": true,
      "retried_at": "2026-10-18T17:12:06Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!lqnH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F995d155d-fbba-4e71-8550-2d3ca482f04c_1200x670.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/009.png",
      "bytes": 273785,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!lqnH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F995d155d-fbba-4e71-8550-2d3ca482f04c_1200x670.png",
      "reused": true,
      "retried_at": "2026-10-18T17:12:06Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!5S35!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36d261be-e556-414b-bd91-312cd5672235_1200x670.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/010.png",
      "bytes": 283007,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!5S35!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36d261be-e556-414b-bd91-312cd5672235_1200x670.png",
      "reused": true,
      "retried_at": "2026-10-18T17:12:06Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!rQqM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd1a5e28-35e9-4222-ad9a-8b40a27f36e7_1200x670.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/011.png",
      "bytes": 337972,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!rQqM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd1a5e28-35e9-4222-ad9a-8b40a27f36e7_1200x670.png",
      "reused": true,
      "retried_at": "2026-10-18T17:12:06Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!j20X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F255f0834-04bc-4beb-bafb-71acbe2be1cc_1199x515.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/012.png",
      "bytes": 224628,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!j20X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F255f0834-04bc-4beb-bafb-71acbe2be1cc_1199x515.png",
      "reused": true,
      "retried_at": "2026-10-18T17:12:06Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!pjSI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4859c-0c00-47bc-a295-a990507c0a8a_911x771.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/013.png",
      "bytes": 107037,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!pjSI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4859c-0c00-47bc-a295-a990507c0a8a_911x771.png",
      "reused": true,
      "retried_at": "2026-10-18T17:12:06Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!ea1r!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9070db6d-1e36-4653-9af3-a4973a1b6af7_1200x670.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/014.png",
      "bytes": 296221,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!ea1r!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9070db6d-1e36-4653-9af3-a4973a1b6af7_1200x670.png",
      "reused": true,
      "retried_at": "2026-10-18T17:12:06Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!pjSI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4859c-0c00-47bc-a295-a990507c0a8a_911x771.png",
      "alt": "",
      "local_path": "attachments/525ad308f1a4/images/013.png",
      "bytes": 107037,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!pjSI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4859c-0c00-47bc-a295-a990507c0a8a_911x771.png",
      "reused": true,
      "retried_at": "2026-10-18T17:12:06Z"
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/p/how-top-ai-labs-are-building-rl-agents-in-2026/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/archive/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/what-is-mcp/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/mlops-crash-course-part-1/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/rl-course-part-1/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/membership"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/p/how-to-beat-grpo-without-touching-model-weights/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/contact/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/faq/"
    },
    {
      "type": "external",
      "href": "https://dailydoseofds.github.io/ai-engg-book/"
    },
    {
      "type": "external",
      "href": "https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view"
    },
    {
      "type": "external",
      "href": "https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view"
    },
    {
      "type": "external",
      "href": "https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view"
    },
    {
      "type": "external",
      "href": "https://github.com/OpenPipe/ART"
    },
    {
      "type": "external",
      "href": "https://x.com/intent/tweet?text=How+Top+AI+Labs+Are+Building+RL+Agents+in+2026&url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-top-ai-labs-are-building-rl-agents-in-2026%2F"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-top-ai-labs-are-building-rl-agents-in-2026%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-top-ai-labs-are-building-rl-agents-in-2026%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/_avichawla"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:25:31Z",
  "source_url": "https://www.dailydoseofds.com/p/how-top-ai-labs-are-building-rl-agents-in-2026/",
  "delivery_status": "unverified",
  "discovery_status": "short_or_dynamic",
  "missing_reason": "正文过短或动态页面，未确认完整",
  "source_body_markdown": "[ Skip to main content  ](https://www.dailydoseofds.com/p/how-top-ai-labs-are-building-rl-agents-in-2026/#sx-main)\n[ ![Daily Dose of Data Science](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/12/dailyds-logo-transparent-1.png) ](https://www.dailydoseofds.com)\n  * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n  * [ Archive ](https://www.dailydoseofds.com/archive/)\n  * Guidebooks \n    * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n[ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n[ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n[ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n[ Courses ](https://www.dailydoseofds.com/)\n[ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n[ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n[ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n[ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n[ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n  * Courses \n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n[ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n[ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n[ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n[ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n[ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n[ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n[ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n  *     * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n[ Archive ](https://www.dailydoseofds.com/archive/)\nGuidebooks \n      * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n      * [ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n      * [ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n      * [ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n      * [ Courses ](https://www.dailydoseofds.com/)\n      * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n      * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n      * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n      * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n      * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\nCourses \n      * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n      * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n      * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n      * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n      * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n      * [ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n      * [ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n      * [ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n\n\nSign In  [ Get Started ](https://www.dailydoseofds.com/membership)\n/\n  * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n  * [ Archive ](https://www.dailydoseofds.com/archive/)\n  * Guidebooks \n    * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n    * [ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n    * [ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n    * [ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n    * [ Courses ](https://www.dailydoseofds.com/)\n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n    * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n    * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n    * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n    * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n  * Courses \n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n    * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n    * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n    * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n    * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n    * [ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n    * [ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n    * [ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n  * Sign In \n\n\nApr 28, 2026  Reinforcement Learning \n#  How Top AI Labs Are Building RL Agents in 2026 \nThe era of not writing custom reward functions. \n![Avi Chawla](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg)\nAvi Chawla \n👉\n* * *\nReinforcement learning, at its core, is straightforward: a system takes an action, the environment rewards it, and the agent updates its behavior to maximize that reward over time.\nThe interaction above works in discrete steps. At each step, three things happen in order:\n![](https://substackcdn.com/image/fetch/$s_!jhge!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccdf09c7-4c52-4478-bda7-6af84f4f9614_1200x670.png)\n  * The agent observes the current state of the environment (S). A state is a description of the situation the agent is in, enough to decide what to do next. For instance, in chess, the state is the board position, and in a dialogue model, the state is the conversation history so far.\n  * The agent picks an action (A) based on what it sees. The action is the agent’s output, the only way it can influence the environment. For instance, in chess, an action is a legal move. For an LLM, an action is the generated response.\n  * The environment then does two things: it transitions to a new state (S’), and it emits a reward (R), a scalar number that evaluates the action. The next step begins, and the loop continues.\n\n\nStringing these steps together gives a trajectory:\n![](https://substackcdn.com/image/fetch/$s_!aFg_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feb0c1dde-10c9-482d-8067-248808dd2c4f_657x165.png)\nReading left to right, this is the entire history of the agent’s interaction with the environment. Each (S, A, R, S’) quartet is one transition, and much of RL is about learning from these transitions.\n### **Applying RL to LLMs**\nWhen RL was first applied to LLMs, the environment was human preference.\nOpenAI’s InstructGPT (2022) introduced RLHF (Reinforcement Learning from Human Feedback), where:\n![](https://substackcdn.com/image/fetch/$s_!zgqX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d3df8df-a431-4346-b758-ef668c61fbd4_1200x670.png)\n  * humans ranked model outputs\n  * those rankings trained a reward model\n  * and PPO (Proximal Policy Optimization) used that reward model to fine-tune the LLM.\n\n\nChatGPT was built on this exact pipeline.\nBut humans can’t sit in the training loop rating every output in real time. If the model generates 16 responses per prompt across thousands of training steps, that’s hundreds of thousands of evaluations.\nOpenAI solved this by splitting the process into two phases.\n![](https://substackcdn.com/image/fetch/$s_!E-qN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2bfdebaf-9d11-4df8-910e-344510a3330e_1200x670.png)\n  * First, the offline phase. Here, humans ranked a relatively small set of model outputs and generated pairwise comparisons. This was the expensive human labor part, but it was a one-time cost.\n  * Second, they trained a reward model on those rankings, which was a separate LLM that learned to predict what humans would prefer. Now you had a neural network that could score any output instantly, without waiting for a human. The reward model was a compressed approximation of human judgment, fast enough to sit inside the training loop.\n\n\nWith the reward model in place, PPO could run the actual RL training at GPU speed. The model generated responses, the reward model scored them, and PPO updated the weights, without extensive need for humans.\nThe cost, however, was that PPO required four full-size models in memory simultaneously.\n![](https://substackcdn.com/image/fetch/$s_!UJA9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3c682ae-d31d-47fb-aa1b-1a8e8d817897_1200x670.png)\n  * The policy (the LLM being trained).\n  * The reference policy (a frozen copy of the original, used to prevent training from drifting too far via a KL divergence penalty).\n  * The reward model (the human-preference approximator discussed above to score every output).\n  * And the critic, also called the value model (more about it below).\n\n\nThe critic exists to answer one question:\n> **Was this reward good or bad relative to what we’d normally expect for this prompt?**\nWe need this because a raw reward of 0.7 means nothing in isolation. For instance, on a simple factual question where most responses score 0.9, a 0.7 is below average.\nBut on a complex open-ended question where most responses score 0.4, a 0.7 is excellent.\n![](https://substackcdn.com/image/fetch/$s_!Cr8C!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5bbf438-b558-4dfc-8efe-0da82db675eb_1200x670.png)\nThe critic learns this baseline by observing thousands of (prompt, reward) pairs during training.\nPPO’s actual training signal is the advantage, which is estimated as the reward minus the critic’s predicted baseline.\nThis makes the signal stable across prompts of different difficulty. But the cost involved here is that the critic is a full-size LLM itself, adding another model’s worth of memory.\nFor a 7B parameter LLM, that meant roughly 28B parameters in memory at once.\n### **DeepSeek R1 breakthrough using verifiable rewards**\nIn January 2025, DeepSeek released R1 with a fundamentally different approach to the reward signal.\nInstead of training a reward model from human preferences (Phases 1 and 2 of the RLHF pipeline), they used RLVR (Reinforcement Learning with Verifiable Rewards).\nIt’s a simple, rule-based verification where the environment itself provides the signal.\n![](https://substackcdn.com/image/fetch/$s_!vKla!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88c192b7-c85b-45ee-8349-ffcf686cd876_1200x651.png)\nFor instance:\n  * For math problems, the verifier checked if the model’s answer matched the known solution.\n  * For code, a compiler ran the output and returned pass or fail. Binary rewards: 1 for correct, 0 for wrong.\n\n\nThere are no human rankings or explicit reward models required since the ground truth was available (or inferable) to be used as the reward.\nThe RL optimizer was GRPO (Group Relative Policy Optimization), which stripped away most of PPO’s infrastructure.\nIt removed the critic model entirely.\nInstead of training a separate model to predict expected reward per prompt, GRPO generated multiple responses to the same prompt (typically 16) and normalized rewards within each group.\n![](https://substackcdn.com/image/fetch/$s_!Z0oW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f2b2be5-8531-4dc9-96e8-d01679216e88_1080x1029.png)\nIf 4 out of 16 responses got the math problem right, those 4 received a positive advantage, and the other 12 received a negative advantage.\nThis step cut an entire full-size model from memory.\nGRPO also removed the need for the learned reward model, since RLVR’s verifier handled scoring directly.\nSo the four-model PPO setup (policy + reference + critic + reward model) collapsed to just two, i.e., the policy being trained and a reference copy for KL regularization.\nIn fact, in practice, some implementations fold the reference into the policy checkpoint, bringing it close to a single-model setup.\nWith this setup. DeepSeek R1-Zero, trained with just GRPO and verifiable rewards (no supervised fine-tuning at all), went from 15.6% to 77.9% on AIME 2024 math problems.\nWith majority voting, it hit 86.7%, matching OpenAI’s o1.\nThe model developed self-verification, reflection, and chain-of-thought reasoning on its own, purely from the binary correct/incorrect signal, and nobody taught it to reason step by step.\nThe RL training loop discovered that reasoning improved the reward, so the model learned to reason.\nRLVR with GRPO became the dominant approach for training reasoning models through 2025.\nEvery major lab released a reasoning variant following this recipe.\n### **The problem**\nGRPO itself is general-purpose.\nIt doesn’t care whether the reward comes from a math verifier, a code compiler, a human, or a Python script.\n![](https://substackcdn.com/image/fetch/$s_!lqnH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F995d155d-fbba-4e71-8550-2d3ca482f04c_1200x670.png)\nIt just needs a number for each response, and it normalizes within each group to produce the training signal.\nBut a clear bottleneck here is where these reward comes from.\nFor math and code, this is fine since the environment provides a deterministic signal.\nBut agents that interact with real-world tools and data don’t produce outputs you can string-match against a gold answer.\nA RAG agent retrieves context and generates a response. There’s no single correct answer to compare against. A customer support agent drafts a reply. There’s no compiler to run it through. A summarization agent condenses a 20-page document. There are many valid summaries, and no string-matching verifier can distinguish a good one from a mediocre one.\n![](https://substackcdn.com/image/fetch/$s_!5S35!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36d261be-e556-414b-bd91-312cd5672235_1200x670.png)\nIn these cases, the environment doesn’t hand you a reward signal the way a math problem does.\nOf course, some agent tasks do have verifiable outcomes, and for these, RLVR works just fine, even with multi-step tool use. The verifiability depends on the task’s outcome, not on whether the model is acting as an agent.\nBut for the majority of agent workflows, the outcome is subjective or multi-dimensional.\nIntuitively, GRPO is still the right fit here because Agents that take multiple steps, call tools, and compose responses would benefit from learning through exploration, trying different approaches, and getting reinforced for what works.\nSo, while the RL framework is the right fit, the missing piece is the scoring function.\nOne solution is to write custom reward functions where Python code scores each output based on hand-defined criteria.\n![](https://substackcdn.com/image/fetch/$s_!rQqM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd1a5e28-35e9-4222-ad9a-8b40a27f36e7_1200x670.png)\n  * A RAG reward function might check whether the response used the retrieved context (faithfulness), penalize content that wasn’t in the context (hallucination), reward completeness, and handle cases where the context itself is ambiguous.\n  * A tool-use reward function might score partial progress through a multi-step task, penalize unnecessary API calls, and measure whether the agent reached the correct final state.\n\n\nEach criterion returns a partial score, and these get summed or weighted into a final reward.\nThis works, but it introduces its own set of problems.\nWriting a good reward function takes days of iteration. Researchers need to anticipate edge cases, calibrate the weights between different criteria, and test that the function actually rewards the behavior you want.\nA reward function that over-weights format compliance and under-weights faithfulness will train an agent that produces beautifully formatted hallucinations.\nReward functions are also brittle. If you change the retrieval pipeline, add a new tool, or modify the system prompt, the reward function needs to be rewritten.\nDebugging is problematic too.\nWhen the agent learns bad behavior during training, the cause could be the reward function, the training hyperparameters, the data, or something else entirely.\nBut because the reward function is custom code, you often can’t tell whether the function is measuring what you think it’s measuring until you’ve already trained a model on it and evaluated the outputs.\nThis is the primary reason RL has been widely adopted for verifiable tasks (math, code, logic) but not for agent workflows (RAG, customer support, tool use, summarization).\nRLVR gave reasoning models a general-purpose, automatic reward signal where they could check the answer and return 0 or 1. No such equivalent exists for most agentic workflows.\nThe distinction isn’t about the model. The same Qwen 2.5 14B can serve both roles.\nThe distinction is about the task. Can we verify if an Agent is producing an output that can be automatically checked?\n### **How are AI labs approaching this?**\nThis isn’t a gap that only open-source practitioners are noticing.\nThe major AI labs have been converging on the same problem from different directions.\nAnthropic demonstrated that you don’t need humans in the RL loop at all.\nTheir Constitutional AI work showed that if you write down a set of principles (a “constitution”), an AI can evaluate outputs against those principles and generate preference data for RL training.\n![](https://substackcdn.com/image/fetch/$s_!j20X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F255f0834-04bc-4beb-bafb-71acbe2be1cc_1199x515.png)\nThe AI judged its own outputs against the written principles and used those judgments as the RL signal. This was a significant conceptual shift that a document of rules replaced an army of human evaluators.\nOpenAI has been working on something similar internally. They are developing “Universal Verifiers,” a technique to extend RL beyond math and code into domains like biology, medicine, and general knowledge, where answers can’t be checked with a simple string match.\nThe details aren’t public, but the direction is clear that we need general-purpose reward signals that work across any domain, not just the ones with deterministic verifiers.\n### **RULER**\nIf you want to see this in practice, RULER, built into [**OpenPipe’s ART framework**](https://github.com/OpenPipe/ART?ref=dailydoseofds.com) (open-source with 9k+ stars) is a general-purpose reward function that replaces all of that custom scoring code with a single function call.\n![](https://substackcdn.com/image/fetch/$s_!pjSI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4859c-0c00-47bc-a295-a990507c0a8a_911x771.png)\nIt uses an LLM-as-judge to rank multiple trajectories, and it works by exploiting the same property that makes GRPO powerful, i.e., only relative rankings matter.\nHere’s how it works step-by-step:\n![](https://substackcdn.com/image/fetch/$s_!ea1r!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9070db6d-1e36-4653-9af3-a4973a1b6af7_1200x670.png)\n  * For each training step, you generate N trajectories for the same scenario (typically 4 to 8).\n  * RULER sends all N to a judge LLM (like o3, o4-mini, or even a local Qwen3 32B).\n  * The judge reads the agent’s system prompt to understand what the agent was supposed to do, then scores each trajectory from 0 to 1 relative to the others.\n\n\nTwo properties make this work:\n**1) Relative scoring is easier than absolute scoring.**\nLLMs struggle with absolute scoring because there’s no shared calibration.\nBut asking “which of these 4 responses best follows the system prompt’s instructions” is a comparison task, and LLMs do those consistently well.\nRULER leans into this by presenting all trajectories together and asking the judge to rank them against each other.\n**2) GRPO normalizes within each group anyway.**\nWhether the best trajectory scored 0.9 or 0.3 in absolute terms doesn’t matter.\nGRPO takes the scores within a group, computes the mean and standard deviation, and normalizes.\nThe training signal comes from the relative ordering by understanding which trajectories were above average and which were below. RULER’s relative rankings map directly onto what GRPO expects.\n### **A rough walkthrough**\nBefore jumping into code, let’s trace what happens conceptually. Say you’re training a RAG agent. At each training step, GRPO generates multiple responses for the same query:\n\n```\nScenario: \"What is the refund policy?\"\nRetrieved context: \"Refunds within 30 days. Digital products non-refundable...\"\n\n(Faithful)\nResponse A: \"Refunds within 30 days. Email support@example.com.\"     \n\n(hallucinated)\nResponse B: \"Refunds within 30 days. Also store credit for 90 days.\"\n\n(ignored context)\nResponse C: \"Not sure, check the website.\"\n\n(verbose but accurate)\nResponse D: \"The policy states that refunds are available within...\"\n```\n\nCopy\nIn a traditional setup, you’d write a reward function to score each of these:\n\n```\ndef reward_function(response, context):\n    score = 0.0\n    if uses_context(response, context):\n        score += 0.4\n    if not has_hallucination(response, context):\n        score += 0.3\n    if is_complete(response, context):\n        score += 0.2\n    if is_concise(response):\n        score += 0.1\n    return score\n```\n\nCopy\nEach of those helper functions (`uses_context`, `has_hallucination`, `is_complete`, `is_concise`) is its own engineering project.\nYou need to define what “uses context” means precisely, decide thresholds, handle edge cases, and test everything.\nWith RULER, you replace all of that with:\n\n```\nscored_group = await ruler_score_group(group, \"openai/o3\")\n```\n\nThe judge LLM reads the system prompt (”Answer using ONLY the retrieved context. Do not add information not in the context.”), reads all four responses and scores them.\nThe system prompt already defines faithfulness, hallucination, and completeness implicitly. The judge applies those criteria without implementing them in Python.\n### **Trajectories and Groups**\nART represents each agent response as a `Trajectory`, and it is a sequence of messages (system, user, assistant) packaged with metadata that GRPO needs for training.\nMultiple trajectories for the same scenario form a `TrajectoryGroup`. This is the unit RULER scores and GRPO trains on.\n\n```\n# A single trajectory: one complete agent interaction\ntraj = art.Trajectory(\n    messages_and_choices=[\n        {\"role\": \"system\", \"content\": \"You are a RAG support agent...\"},\n        {\"role\": \"user\", \"content\": \"What is the refund policy?\\n\\n[Context]: ...\"},\n        Choice(finish_reason=\"stop\", index=0,\n               message=ChatCompletionMessage(role=\"assistant\", content=\"...\")),\n    ],\n    reward=0.0,  # RULER fills this in\n)\n\n# A group: multiple trajectories for the same scenario\ngroup = art.TrajectoryGroup([traj_a, traj_b, traj_c, traj_d])\n\n# Score the entire group relatively\njudged_group = await ruler_score_group(group, \"openai/o3\")\n```\n\nCopy\nThe `reward=0.0` initialization is a placeholder.\n  * After `ruler_score_group` returns, each trajectory’s reward field is updated with the judge’s score.\n  * The `Choice` and `ChatCompletionMessage` objects are OpenAI’s standard types, so if you’re already using the OpenAI SDK for inference, these are the same objects you’re already working with.\n\n\n### **Two concrete examples**\nRULER has two levels of API.\n1) The lower-level `ruler` function works with plain message dictionaries, so no ART-specific objects are needed.\nThis is the fastest way to see what RULER actually does.\nLet’s look at a minimal example with three trajectories for the same RAG query.\nBelow, we have defined three message lists: one is faithful, one hallucinates, and one ignores the context entirely.\n\n```\nmessage_lists = [\n    # Good: uses retrieved context accurately\n    [\n        {\"role\": \"system\", \"content\": \"You are a RAG-based support agent. Use the retrieved context to answer user queries accurately.\"},\n        {\"role\": \"user\", \"content\": \"What is our refund policy?\\n\\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate.\"},\n        {\"role\": \"assistant\", \"content\": \"You can get a refund within 30 days of purchase. However, digital products are non-refundable once downloaded. To start the process, email support@example.com.\"},\n    ],\n    # Mediocre: partially uses context, adds hallucinated details\n    [\n        {\"role\": \"system\", \"content\": \"You are a RAG-based support agent. Use the retrieved context to answer user queries accurately.\"},\n        {\"role\": \"user\", \"content\": \"What is our refund policy?\\n\\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate.\"},\n        {\"role\": \"assistant\", \"content\": \"Our refund policy allows returns within 30 days. You can also get store credit for up to 90 days. Just call our hotline at 1-800-REFUND.\"},\n    ],\n    # Bad: ignores context entirely\n    [\n        {\"role\": \"system\", \"content\": \"You are a RAG-based support agent. Use the retrieved context to answer user queries accurately.\"},\n        {\"role\": \"user\", \"content\": \"What is our refund policy?\\n\\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate.\"},\n        {\"role\": \"assistant\", \"content\": \"I'm not sure about the refund policy. You should check the website or contact someone from the billing team.\"},\n    ],\n]\n```\n\nCopy\nNext, we run the scoring:\n\n```\nfrom art.rewards import ruler\n\nscores = await ruler(message_lists, \"openai/o3\")\n\nfor label, score in zip([\"Faithful\", \"Hallucinated\", \"Ignored context\"], scores):\n    print(label)\n    print(\"→\", score.score)\n    print(\"→\", score.explanation)\n```\n\nCopy\nThis produces the following output:\n\n```\nFaithful:\n→ 0.97 \n→ Accurately reflects the retrieved policy\n  details, complete and concise.\n\nHallucinated:\n→ 0.45\n→ Gives correct 30-day refund info but adds\n  unsupported details (90-day credit, hotline),\n  reducing accuracy.\n\nIgnored context:\n→ 0.05\n→ Provides no useful information and ignores available context.\n```\n\nCopy\nNotice that we never wrote a faithfulness checker or coded a hallucination detector.\nThe system prompt mentioned “Use the retrieved context to answer user queries accurately,” and the judge applied that as the evaluation criteria.\nThe hallucinated response scored 0.45 (not zero) because it partially used the context. The 30-day refund part was correct.\nThe judge gave partial credit for what it got right and penalized what it invented.\nThat’s a nuanced distinction that would take significant engineering to encode in a rule-based reward function.\nMoreover, the scores are spread across the 0-1 range: 0.97, 0.45, 0.05, unlike binary pass/fail.\nRULER produces a gradient that reflects relative quality. GRPO can use this gradient to apply proportional updates to strongly reinforce the faithful behavior, mildly suppress the hallucination pattern (since it was partially correct), and strongly suppress the context-ignoring behavior.\n2) The `ruler` function above works for understanding and experimentation, but ART’s training loop operates on `Trajectory` and `TrajectoryGroup` objects.\nThese carry the reward field that GRPO reads, debug logs for inspection, and the structure that `model.train()` expects.\nAfter this, the higher-level `ruler_score_group` function handles the conversion.\nBelow, let’s look at the same RAG scenario structured the way you’d use it in a real training pipeline, now with 4 trajectories instead of 3.\n\n```\n# The system prompt defines the agent's goal\n# RULER uses this as the implicit reward function\nsystem_msg = {\n    \"role\": \"system\",\n    \"content\": (\n        \"You are a RAG-based support agent. \"\n        \"Answer user queries using ONLY the retrieved context. \"\n        \"Do not add information that is not in the context.\"\n    ),\n}\n\nuser_msg = {\n    \"role\": \"user\",\n    \"content\": (\n        \"What is the refund policy?\\n\\n\"\n        \"[Retrieved context]: Refunds are available within 30 days \"\n        \"of purchase. Digital products are non-refundable after \"\n        \"download. Contact support@example.com to initiate.\"\n    ),\n}\n\nresponses = [\n    \"You can get a refund within 30 days of purchase. Digital products \"\n    \"are non-refundable once downloaded. Email support@example.com to start.\",\n\n    \"Refunds are available within 30 days. You can also get store credit \"\n    \"for up to 90 days, and our hotline is 1-800-REFUND.\",\n\n    \"I'm not sure about the refund policy. Please check the website or \"\n    \"contact the billing team for more details.\",\n\n    \"Based on the information I have, the refund policy states that \"\n    \"refunds are available within 30 days of purchase. It is important \"\n    \"to note that digital products cannot be refunded after they have \"\n    \"been downloaded. If you wish to initiate a refund, you should \"\n    \"reach out to support@example.com.\",\n]\n```\n\nCopy\nNow we have 4 trajectories instead of 3. The fourth is a verbose but accurate response that uses only the retrieved context but wraps it in unnecessary filler words/sentences.\nMoving on, we define our `Trajectories` and `Groups` as we discussed earlier:\n\n```\nimport art\nfrom openai.types.chat.chat_completion import Choice\nfrom openai.types.chat import ChatCompletionMessage\n\ntrajectories = []\nfor resp in responses:\n    traj = art.Trajectory(\n        messages_and_choices=[\n            system_msg, user_msg,\n            Choice(\n                finish_reason=\"stop\", index=0,\n                message=ChatCompletionMessage(role=\"assistant\", content=resp),\n            ),\n        ],\n        reward=0.0,\n    )\n    trajectories.append(traj)\n\ngroup = art.TrajectoryGroup(trajectories)\n```\n\nCopy\nFinally, we run the scoring:\n\n```\nfrom art.rewards import ruler_score_group\n\njudged_group = await ruler_score_group(group, \"openai/o3\", debug=True)\n```\n\nCopy\nWith `debug=True`, RULER prints the judge’s raw reasoning with the actual scores.\nThis is the raw reasoning:\n\n```\n{\n    \"scores\": [\n        {\n            \"trajectory_id\": \"1\",\n            \"explanation\": \"Accurately answers the question using only the retrieved context, concisely and completely.\",\n            \"score\": 0.98\n        },\n        {\n            \"trajectory_id\": \"2\",\n            \"explanation\": \"Includes unsupported details about store credit and a hotline that are not in the retrieved context, so it violates the instruction to use only the context.\",\n            \"score\": 0.2\n        },\n        {\n            \"trajectory_id\": \"3\",\n            \"explanation\": \"Does not answer the question despite the needed information being present in the retrieved context.\",\n            \"score\": 0.05\n        },\n        {\n            \"trajectory_id\": \"4\",\n            \"explanation\": \"Accurately and completely answers the question using only the retrieved context, though slightly more verbose than necessary.\",\n            \"score\": 0.96\n        }\n    ]\n}\n```\n\nCopy\nAnd these are the scores (ranked):\n\n```\nRank 1 | Score: 0.980 — Concise, faithful response\nRank 2 | Score: 0.960 — Verbose but accurate response\nRank 3 | Score: 0.200 — Hallucinated store credit and hotline\nRank 4 | Score: 0.050 — Ignored the retrieved context entirely\n```\n\nCopy\nIf you notice closely...\n  * The concise faithful response (0.98) scored just above the verbose accurate one (0.96). Both used only the retrieved context, both were correct, but the system prompt said “Answer using ONLY the retrieved context,” and the concise version did that more directly. The judge recognized verbosity as a minor quality issue, not a correctness issue. That’s a nuanced distinction that would be hard to encode in a scoring function because how do you write a rule that says “technically correct but unnecessarily wordy, penalize by 0.02”?\n  * The hallucinated response dropped from 0.45 in the first experiment to 0.20 here. The difference is the system prompt. The first experiment said “Use the retrieved context to answer accurately.” This one says “Do not add information that is not in the context.” The stricter instruction produced stricter scoring. The judge adapted automatically. If you tighten your system prompt, RULER tightens its evaluation to match, without you changing any scoring code.\n  * The context-ignoring response scored 0.05 in both experiments. When the answer is right there in the retrieved context, and the agent says “I’m not sure,” there’s no ambiguity regardless of how the system prompt is worded.\n\n\nThese scored trajectories are exactly what `model.train()` expects, so let’s look at that ahead.\n### **The full training loop**\nTo actually train with these scores, you replace the hardcoded responses with real model inference.\nART’s `gather_trajectory_groups` handles the orchestration.\nEssentially, for each scenario, it generates a group of trajectories using the model’s current weights, scores them with RULER, and collects the results for GRPO:\n\n```\nfor step in range(num_steps):\n    groups = await art.gather_trajectory_groups(\n        (\n            art.TrajectoryGroup(\n                rollout(model, scenario) for _ in range(4)\n            )\n            for scenario in scenarios\n        ),\n        after_each=lambda g: ruler_score_group(\n            g, \"openai/o3\"),\n    )\n    await model.train(groups)  # GRPO updates LoRA weights\n```\n\nCopy\nIn every step, the model generates 4 responses per scenario using its current weights, RULER ranks them relatively, and GRPO reinforces the high-scoring behavior while suppressing the low-scoring behavior.\nThe agent gets better at following the system prompt’s instructions with every iteration.\nOver multiple steps, the model learns the patterns that score well (faithfulness, conciseness, grounding in context) and unlearns the patterns that score poorly (hallucination, ignoring context, verbosity).\nAnd notice that no reward function was defined anywhere in this code.\n### **Custom rubrics**\nFor most tasks, the system prompt provides enough signal for RULER to score effectively. But when you need more specific evaluation criteria, RULER supports custom rubrics:\n\n```\ncustom_rubric = \"\"\"\n- Prioritize responses that are concise and clear\n- Penalize responses that include emojis or informal language\n- Reward responses that cite sources\n\"\"\"\n\nawait ruler_score_group(group, \"openai/o3\", rubric=custom_rubric)\n```\n\nCopy\nThe rubric is natural language, not Python, so iterating on it is fast.\nYou just change a sentence, rerun, and check the scores.\nCompare this to editing a reward function where a misplaced weight or a buggy condition can silently teach the agent bad behavior that you won’t notice until after training.\n### **Application to non-verifiable tasks**\nRULER is general-purpose. It works on any task, not just freeform ones where custom rewards are painful.\nThe practical question is when RULER adds value over simpler alternatives.\nFor purely deterministic tasks (did the SQL query return the right rows?), a binary verifier is cheaper and gives a cleaner signal.\nFor purely subjective tasks (was the summary good?), RULER is the only automatic option. For tasks that sit in between (did the agent find the right answer AND explain it well?), you can combine both:\n\n```\njudged_group = await ruler_score_group(group, \"openai/o3\")\n\nfor traj in judged_group.trajectories:\n    independent_reward = verify_correctness(traj)  # binary 0/1\n    traj.reward += independent_reward\n```\n\nCopy\nRULER preserves any rewards you assign during rollout under a separate metric, so you can layer LLM-judge scoring on top of deterministic verification without losing either signal.\n### **Practical details**\nHere are some practical insights we have gathered based on using RULER:\n  * You don’t need the most expensive model as the judge. Cheaper models like Qwen3 32B often work well. You can also use Claude, local models through Ollama, or any model supported by LiteLLM. The choice is a cost-quality tradeoff, not a hard requirement.\n  * 4 to 8 trajectories per group is the recommended range. Fewer than 4 gives the judge too little to compare against. More than 8 can confuse the judge and increase the cost without proportional benefit.\n  * When all trajectories in a group share the same system prompt and user message (which they usually do), RULER deduplicates the common prefix automatically. The judge only sees the shared context once, followed by the different responses. This cuts token usage significantly for long system prompts or multi-turn conversations.\n  * RULER caches judge responses to disk. If you rerun the same trajectories, it won’t hit the API again. This matters during debugging when you’re iterating on the system prompt or rubric.\n\n\nThe bottleneck in applying RL to agents was never the optimization algorithm.\nGRPO handles that well.\nIt was always the reward signal.\nRLVR solved this for verifiable tasks by letting the environment score outputs directly.\nRULER solves it for every task (verifiable or non-verifiable) by letting an LLM judge score outputs relatively.\nThe full implementation is in the ART repository, along with Colab notebooks that walk you through the training loop end-to-end.\nRepo: [**https://github.com/OpenPipe/ART**](https://github.com/OpenPipe/ART?ref=dailydoseofds.com) (don’t forget to star it ⭐️)\n![](https://substackcdn.com/image/fetch/$s_!pjSI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4859c-0c00-47bc-a295-a990507c0a8a_911x771.png)\nThanks for reading!\nPublished on Apr 28, 2026 \nComments \nShare \nCopy link [ Share to X ](https://x.com/intent/tweet?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-top-ai-labs-are-building-rl-agents-in-2026%2F&text=How%20Top%20AI%20Labs%20Are%20Building%20RL%20Agents%20in%202026) [ Share to Facebook ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-top-ai-labs-are-building-rl-agents-in-2026%2F) [ Share to Linkedin ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-top-ai-labs-are-building-rl-agents-in-2026%2F)\nCopied \n[ Previous  Foundations of Reinforcement Learning  ](https://www.dailydoseofds.com/rl-course-part-1/) [ Next  How to Beat GRPO Without Touching Model Weights  ](https://www.dailydoseofds.com/p/how-to-beat-grpo-without-touching-model-weights/)\n[ ![Daily Dose of Data Science](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/12/dailyds-logo-transparent-1.png) ](https://www.dailydoseofds.com)\nA daily column with insights, observations, tutorials and best practices on python and data science. Read by industry professionals at big tech, startups, and engineering students. \n  * Menu \n  * [ Contact ](https://www.dailydoseofds.com/contact/)\n  * [ FAQ ](https://www.dailydoseofds.com/faq/)\n\n\n[Daily Dose of Data Science](https://www.dailydoseofds.com) © 2026 \n[ ](https://x.com/_avichawla \"X \\(Formerly Twitter\\)\")\n",
  "body_base_path": "attachments/525ad308f1a4/pages/",
  "body_markdown_path": "attachments/525ad308f1a4/pages/archived-body.md",
  "local_html": "attachments/525ad308f1a4/pages/archived-body.html",
  "body_source": "prior_capture"
}