{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Aleksei Romanov",
  "title_listed": "Multi-Reward Reinforcement Learning for LLM Agents: Comparing PPO, GRPO, DAPO, and GDPO",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/g_factor/multi-reward-reinforcement-learning-for-llm-agents-comparing-ppo-grpo-dapo-and-gdpo-jaf",
  "publish_date": "2026-09-25",
  "corpus_stage": "",
  "notes": "",
  "id": "dd14311b5280",
  "collected_at": "2026-10-02T13:54:39",
  "title": "Multi-Reward Reinforcement Learning for LLM Agents: Comparing PPO, GRPO, DAPO, and GDPO - DEV Community",
  "body_markdown": "New empirical follow-up:\nPart 2 compares seven trainer configurations on Qwen3-14B and our DEX gym\n, with no-think and thinking holdouts, interactive reward curves, and downloadable data. It is a separate experiment from the 27B table below and does not establish a universal trainer ranking.\nIf you only ever train language models on toy math puzzles, reinforcement learning feels simple: did the model output 42? If yes, reward is 1. If no, reward is 0.\nThe moment you try to train an autonomous agent for real-world enterprise work, however,\na single scalar reward is an absolute illusion\n.\nIn production, your agent has to juggle multiple competing, messy, non-commensurate priorities at the same time:\nThe Main Mission (R₁):\nDid it actually solve the customer’s request? (e.g., execute the right SQL query, compile the circuit, return the right data payload).\nExecution Efficiency (R₂):\nDid it solve it elegantly in 3 tool calls, or did it run a wild 40-step loop that burned $4 in API tokens and spiked database CPU?\nHard Guardrails & Constraints (R₃):\nDid it stay inside the sandbox? Did it strictly adhere to output JSON schemas, avoid mutating production tables, and preserve security invariants?\nHere is the dirty secret of post-training: if you take these three scores and simply add them together into standard algorithms like PPO or vanilla GRPO,\nyour training run will almost certainly tear itself apart\n. The loudest reward channel swallows the subtle ones, the agent learns to game the system, and up to a third of your expensive GPU batches end up generating zero gradients.\nStart here\nUnderstand how several distinct reward channels are combined into policy advantages, why simple weighted sums cause catastrophic scale dominance, how GDPO decouples channel normalization, and why hard constraints require enforcement rather than reward weights.\nPolicy:\nThe neural network that chooses the next token or action given the current context.\nRollout:\nOne complete candidate attempt at a task, including intermediate tool calls and final answers.\nReward channel:\nAn independent score assigned to an attempt by a specific evaluator (e.g. task success, latency, format).\nAdvantage:\nA normalized scalar indicating how much better or worse an attempt was compared to its baseline.\nScale dominance:\nWhen a high-variance reward channel mathematically drowns out subtle constraint metrics during joint normalization.\n1. The Algorithmic Evolution: PPO → GRPO → DAPO → GDPO\nTo understand why modern multi-reward agent post-training looks the way it does, we must trace how policy gradient estimators evolved:\n1.1 PPO: The VRAM-Hungry Workhorse\nProximal Policy Optimization (Schulman et al., 2017) was the engine behind the original RLHF revolution. It uses an Actor-Critic architecture: the\nActor\ngenerates the tokens, and a separate\nCritic\nnetwork learns to predict the expected future reward from state\ns\n.\nA\nt\nG\nA\nE\n=\n∑\nl\n=\n0\n∞\n(\nγ\nλ\n)\nl\nδ\nt\n+\nl\nV\n,\nwhere\nδ\nt\nV\n=\nr\nt\n+\nγ\nV\nϕ\n(\ns\nt\n+\n1\n)\n−\nV\nϕ\n(\ns\nt\n)\nA_t^{\\mathrm{GAE}} = \\sum_{l=0}^{\\infty} (\\gamma \\lambda)^l \\delta_{t+l}^V,\\quad \\text{where}\\quad \\delta_t^V = r_t + \\gamma V_\\phi(s_{t+1}) - V_\\phi(s_t)\nA\nt\nGAE\n​\n=\nl\n=\n0\n∑\n∞\n​\n(\nγλ\n)\nl\nδ\nt\n+\nl\nV\n​\n,\nwhere\nδ\nt\nV\n​\n=\nr\nt\n​\n+\nγ\nV\nϕ\n​\n(\ns\nt\n+\n1\n​\n)\n−\nV\nϕ\n​\n(\ns\nt\n​\n)\nWhy PPO Hurts in Practice:\nThe VRAM Double-Tax:\nIf your policy is a 27B model, your Critic is usually another 27B model. You have to hold two massive models in GPU memory along with their optimizer states and activations. You end up needing twice as many H100s just to keep the critic alive.\nCritic Drift on Multi-Reward Tasks:\nTrying to train a single critic head to predict a composite stew of task accuracy, latency penalties, and format compliance is notoriously unstable. The critic gets confused, advantages get noisy, and policy updates turn sluggish.\n1.2 GRPO: Ditching the Critic Entirely\nDeepSeekMath (2024) introduced Group Relative Policy Optimization (GRPO), and it felt like a breath of fresh air. GRPO tossed the Critic network into the recycling bin.\nInstead of asking a neural net to predict a baseline, GRPO samples a group of\nG\ncandidate completions {o₁, o₂, ..., o_G} for the same prompt, scores them all, and normalizes advantages against the group’s own mean and standard deviation:\nA\ni\n=\nR\ni\n−\nmean\n⁡\n(\n{\nR\n1\n,\n…\n,\nR\nG\n}\n)\nstd\n⁡\n(\n{\nR\n1\n,\n…\n,\nR\nG\n}\n)\n+\nϵ\nA_i = \\frac{R_i - \\operatorname{mean}(\\{R_1, \\dots, R_G\\})}{\\operatorname{std}(\\{R_1, \\dots, R_G\\}) + \\epsilon}\nA\ni\n​\n=\nstd\n({\nR\n1\n​\n,\n…\n,\nR\nG\n​\n})\n+\nϵ\nR\ni\n​\n−\nmean\n({\nR\n1\n​\n,\n…\n,\nR\nG\n​\n})\n​\nInstant win: GPU memory needs dropped in half! But when applied to multi-objective environments, vanilla GRPO made an innocent-looking mathematical assumption called\nSum-then-Normalize\n:\nR\ni\n=\n∑\nk\n=\n1\nK\nw\nk\nr\ni\n,\nk\n,\nA\ni\n=\nR\ni\n−\nμ\nR\nσ\nR\n+\nϵ\nR_i=\\sum_{k=1}^{K}w_k r_{i,k},\\qquad A_i=\\frac{R_i-\\mu_R}{\\sigma_R+\\epsilon}\nR\ni\n​\n=\nk\n=\n1\n∑\nK\n​\nw\nk\n​\nr\ni\n,\nk\n​\n,\nA\ni\n​\n=\nσ\nR\n​\n+\nϵ\nR\ni\n​\n−\nμ\nR\n​\n​\nAs we will demonstrate below, this single line of math creates a devastating failure mode:\nScale Dominance\n.\n1.3 DAPO: Rescuing Dead Groups with Dynamic Sampling\nWhen you run GRPO on difficult engineering problems, you quickly discover the curse of\nDead Groups\n. If a coding task is tough and all 8 candidate rollouts in a group fail with a syntax error, every single rollout gets a reward of 0.\nWhen all rewards are 0, the group standard deviation is 0. That means the advantage is 0 across the entire group! Your expensive GPU cluster just spent 30 seconds generating tokens, and the gradient update is completely empty. In hard tasks,\n30% to 40% of all training steps can be dead groups\n.\nDAPO introduced\ndynamic sampling\n: during rollout scoring, if a group has zero variance, it immediately discards the dead data and pulls fresh active prompts until every training batch contains real learning signal, cutting wasted GPU cycles to near zero.\nA group where all 8 attempts fail has zero variance and gives no gradient. DAPO's dynamic sampling replaces it with a prompt whose attempts differ.\n1.4 GDPO: Decoupled Normalization (Normalize-then-Sum)\nIntroduced in 2026 (arXiv:2601.05242) and integrated into modern libraries like TRL 1.7,\nGDPO\nfixes the fundamental multi-reward flaw of GRPO. Instead of adding raw scores together and then normalizing, GDPO enforces\nNormalize-then-Sum\n:\nz\ni\n,\nk\n=\nr\ni\n,\nk\n−\nμ\nk\nσ\nk\n+\nϵ\ns\ni\n=\n∑\nk\n=\n1\nK\nw\nk\nz\ni\n,\nk\nA\ni\n=\ns\ni\n−\nμ\nb\na\nt\nc\nh\nσ\nb\na\nt\nc\nh\n+\nϵ\n\\begin{aligned}z_{i,k}&=\\frac{r_{i,k}-\\mu_k}{\\sigma_k+\\epsilon}\\\\s_i&=\\sum_{k=1}^K w_k z_{i,k}\\\\A_i&=\\frac{s_i-\\mu_{\\mathrm{batch}}}{\\sigma_{\\mathrm{batch}}+\\epsilon}\\end{aligned}\nz\ni\n,\nk\n​\ns\ni\n​\nA\ni\n​\n​\n=\nσ\nk\n​\n+\nϵ\nr\ni\n,\nk\n​\n−\nμ\nk\n​\n​\n=\nk\n=\n1\n∑\nK\n​\nw\nk\n​\nz\ni\n,\nk\n​\n=\nσ\nbatch\n​\n+\nϵ\ns\ni\n​\n−\nμ\nbatch\n​\n​\n​\nEvery reward channel is normalized independently across the group first. Now, whether a reward channel naturally varies between [0, 1.0] or between [0, 0.05], both channels have a mean of 0 and a variance of 1. The small metric can no longer be bullied by the large one.\nSum-then-normalize (GRPO) lets the high-variance task reward drown out token efficiency; normalize-then-sum (GDPO) puts both channels on the same scale before weighting.\n2. Worked Example: Does Changing Score Units Change the Learning Signal?\nSuppose attempts A and B are correct, while C and D are wrong. Each also receives an efficiency score: B is both correct and efficient. C is wrong but very fast—returning an incorrect constant can be lightning-fast.\nThe figure below compares\nSum, then normalize\nwith\nNormalize each, then sum\n, and reports efficiency on a 0–1 and on a 0–100 scale:\nOne prompt, four attempts, equal weights. Summing raw scores lets a change of units (efficiency × 100) hand the top advantage to a wrong answer; normalizing each channel first does not depend on units.\nThe\noriginal post\nhas an interactive version of this example.\nNotice what happens under joint summation: when efficiency is reported on a 0–100 scale, the incorrect but fast candidate C suddenly receives a higher positive advantage than the correct solution! Under decoupled normalization, candidate B remains the standout winner regardless of unit scales.\n3. The Mathematics of Scale Dominance: Why Simple Sums Fail\nTo see why Sum-then-Normalize fails mathematically, look at the variance of a sum of two independent reward signals:\nVar\n⁡\n(\nR\n)\n=\nVar\n⁡\n(\nR\n1\n)\n+\nVar\n⁡\n(\nR\n2\n)\n+\n2\nCov\n⁡\n(\nR\n1\n,\nR\n2\n)\n\\operatorname{Var}(R) = \\operatorname{Var}(R_1) + \\operatorname{Var}(R_2) + 2\\operatorname{Cov}(R_1, R_2)\nVar\n(\nR\n)\n=\nVar\n(\nR\n1\n​\n)\n+\nVar\n(\nR\n2\n​\n)\n+\n2\nCov\n(\nR\n1\n​\n,\nR\n2\n​\n)\nIn an agent task:\nChannel 1 (Task Success) is binary:\nr₁\n∈ {0, 1}. Its variance is roughly σ₁² ≈ 0.25.\nChannel 2 (Token Efficiency) is small:\nr₂\n∈ [0, 0.05]. Its variance is tiny: σ₂² ≈ 0.0006.\nWhen you sum them up, Channel 1 accounts for\n99.7% of the total variance\n. When GRPO divides by σ\nR\n, Channel 2 is effectively multiplied by zero. The model quickly learns a toxic heuristic:\n“I can spam 50 unnecessary tool calls and burn huge token budgets, because the efficiency penalty is mathematically invisible to my gradients!”\n4. Geometric Reward Collapse: When Winners Look Like Losers\nUnder vanilla GRPO, distinct trade-offs get mashed into the exact same scalar score:\nCandidate A:\nPerfect task solution (\nR₁\n= 1.0), but horribly violates formatting rules (\nR₃\n= 0.0). Total = 1.0.\nCandidate B:\nFlawless format and safe execution (\nR₃\n= 0.2), but solved 80% of the core task (\nR₁\n= 0.8). Total = 1.0.\nTo vanilla GRPO, both candidates look identical (\nA\nA\n= A\nB\n). The policy receives zero gradient to choose the clean, compliant solution over the broken, unsafe one. With GDPO’s decoupled normalization, Candidate B’s excellence on the constraint channel stands out with a strong positive sub-advantage, guiding the model toward the true Pareto frontier.\n5. Hard Telemetry from gft-studio\nWe tested all four algorithms on an identical multi-objective agent gym in\ngft-studio\n, evaluating task success (\nR₁\n), token efficiency (\nR₂\n), and constraint compliance (\nR₃\n) on a 27B model:\nMethod & Normalization\nCritic Model?\nComposite Pareto Reward\nTask Success Rate\nConstraint Violation Rate\nDead Group Rate\nPPO (Actor-Critic Baseline)\nYes (27B Critic, GAE)\n0.312 ± 0.04\n42.5%\n24.8%\nN/A (GAE baseline)\nGRPO (Vanilla Joint Sum)\nNo (Critic-Free)\n0.395 ± 0.03\n51.2%\n29.4%\n34.2% (Dead Groups)\nDAPO (Dynamic Refill + GRPO)\nNo (Critic-Free)\n0.448 ± 0.02\n56.8%\n22.1%\n< 3.0% (Refilled)\nGDPO (Decoupled Norm + Safety)\nNo (Critic-Free)\n0.562 ± 0.02\n63.4%\n< 3.8%\n< 3.0% (Refilled)\nThe Big Takeaways\nVanilla GRPO Bleeds Safety Constraints:\nWhile it solved the primary task 51.2% of the time, its constraint violation rate was alarming:\n29.4%\n. Because task reward dominated the sum, the agent routinely broke formatting and safety contracts to get the job done.\nDAPO Saves 34% of Wasted Compute:\nOver a third of vanilla GRPO groups had zero variance. DAPO’s dynamic prompt replenishment ensured that every batch drove meaningful parameter updates.\nGDPO Nails the Pareto Frontier:\nBy standardizing each reward channel independently, constraint violations collapsed from 29.4% down to\nunder 3.8%\n, while overall task success jumped to\n63.4%\n.\n6. The Practitioner’s Field Guide\nWhen building reinforcement learning pipelines for real-world enterprise agents:\nNever use joint summation for multi-reward environments.\nAlways normalize channels independently first (Normalize-then-Sum).\nHard constraints belong in the tool interface, not in soft weights.\nIf an agent must never write unauthorized data to production, block the write at the API barrier. Do not rely solely on negative rewards to enforce safety.\nTurn on dynamic group sampling (DAPO) if your task is hard.\nIf your baseline model solves the problem less than 20% of the time, you will waste huge amounts of GPU compute on dead groups without it.\nClamp your combined advantages.\nWhen combining multiple normalized channels, an outlier rollout that spikes on two channels simultaneously can cause massive policy drift. Always apply a safety clamp (\nc\nmax\n∈ [3.0, 5.0]) to protect training stability.\nApplying this to your own models?\nWe build private RL training environments with deterministic verifiers for teams' own workflows.\nPrivate training environments\n.\nOriginally published at\ng-ftech.com\n.",
  "metadata": {
    "title": "Multi-Reward Reinforcement Learning for LLM Agents: Comparing PPO, GRPO, DAPO, and GDPO - DEV Community",
    "description": "GDPO vs GRPO, DAPO and PPO for multi-reward agent post-training: normalization math, scale dominance, reward collapse, and benchmark results. Tagged with machinelearning, ai, llm, reinforcementlearning.",
    "keywords": "machinelearning, ai, llm, reinforcementlearning, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/g_factor/multi-reward-reinforcement-learning-for-llm-agents-comparing-ppo-grpo-dapo-and-gdpo-jaf",
    "og:title": "Multi-Reward Reinforcement Learning for LLM Agents: Comparing PPO, GRPO, DAPO, and GDPO",
    "og:description": "GDPO vs GRPO, DAPO and PPO for multi-reward agent post-training: normalization math, scale dominance, reward collapse, and benchmark results.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fwww.g-ftech.com%2Fog%2Fmulti-reward-rl-ppo-grpo-dapo-gdpo.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Multi-Reward Reinforcement Learning for LLM Agents: Comparing PPO, GRPO, DAPO, and GDPO",
    "twitter:description": "GDPO vs GRPO, DAPO and PPO for multi-reward agent post-training: normalization math, scale dominance, reward collapse, and benchmark results.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fwww.g-ftech.com%2Fog%2Fmulti-reward-rl-ppo-grpo-dapo-gdpo.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4tglgo8d0164n94sep3d.png",
      "bytes": 92064,
      "reused": true,
      "local_path": "attachments/dd14311b5280/images/003.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4tglgo8d0164n94sep3d.png",
      "alt": "A group where all 8 attempts fail has zero variance and gives no gradient. DAPO's dynamic sampling replaces it with a prompt whose attempts differ.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsorxosixiixufz4nr05y.png",
      "bytes": 72768,
      "reused": true,
      "local_path": "attachments/dd14311b5280/images/004.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsorxosixiixufz4nr05y.png",
      "alt": "Sum-then-normalize (GRPO) lets the high-variance task reward drown out token efficiency; normalize-then-sum (GDPO) puts both channels on the same scale before weighting.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvqi8dwwwk0v8f5d75lr6.png",
      "bytes": 80700,
      "reused": true,
      "local_path": "attachments/dd14311b5280/images/005.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvqi8dwwwk0v8f5d75lr6.png",
      "alt": "One prompt, four attempts, equal weights. Summing raw scores lets a change of units (efficiency × 100) hand the top advantage to a wrong answer; normalizing each channel first does not depend on units.",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.g-ftech.com/blog/multi-reward-rl-part-2-trainer-benchmark?utm_source=devto&utm_medium=syndication",
      "label": "Part 2 compares seven trainer configurations on Qwen3-14B and our DEX gym",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4tglgo8d0164n94sep3d.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsorxosixiixufz4nr05y.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvqi8dwwwk0v8f5d75lr6.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.g-ftech.com/blog/multi-reward-rl-ppo-grpo-dapo-gdpo?utm_source=devto&utm_medium=syndication",
      "label": "original post",
      "type": "a"
    },
    {
      "href": "https://www.g-ftech.com/solutions/private-training-environments?utm_source=devto&utm_medium=syndication",
      "label": "Private training environments",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:57Z",
  "source_url": "https://dev.to/g_factor/multi-reward-reinforcement-learning-for-llm-agents-comparing-ppo-grpo-dapo-and-gdpo-jaf",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 12312,
  "expected_body_images": 3,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/dd14311b5280/pages/source.html",
  "local_html": "attachments/dd14311b5280/pages/article.html",
  "body_base_path": "attachments/dd14311b5280/pages/",
  "body_markdown_path": "attachments/dd14311b5280/pages/article.md"
}