{ "source_file": "0930", "source_platform": "Dev.to", "author": "Arun Kumar", "title_listed": "GRPO doesn't remove the reward model. It removes the critic.", "keywords": "", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://dev.to/narotra05hp/grpo-doesnt-remove-the-reward-model-it-removes-the-critic-2pnp", "publish_date": "2026-09-18", "corpus_stage": "", "notes": "", "id": "80ba6e9d1515", "collected_at": "2026-10-02T13:54:51", "title": "GRPO doesn't remove the reward model. It removes the critic. - DEV Community", "body_markdown": "Every time GRPO comes up I see the same slip — someone says it \"gets rid of the reward model\". It doesn't. It gets rid of the\nvalue\nmodel. Those are two different networks doing unrelated jobs, and telling them apart is most of understanding the algorithm.\nWorth going back to what the DeepSeekMath paper actually claims, because it's more modest than the version that travels.\nIt's a variant, not a successor\nShao and colleagues introduce GRPO as \"a variant of Proximal Policy Optimization (PPO)\". Not a replacement. A variant. It keeps PPO's clipped ratio, and it keeps the reference model. Three things change: the critic goes, the baseline becomes group-relative, and the KL term moves.\nThat's narrower than \"GRPO replaces PPO\", which is how I usually see it written up.\nWhat the critic was costing\nPPO is actor-critic, so the critic gets trained alongside the policy. The paper is blunt about the bill. That value function is \"typically another model of comparable size as the policy model\", and so \"it brings a substantial memory and computational burden\".\nYou're training two large models to ship one.\nThere's a second complaint I found more interesting than the memory one. In language-model RL, \"usually only the last token is assigned a reward score by the reward model\". A signal that sparse \"may complicate the training of a value function that is accurate at each token\". So the critic is expensive\nand\nhard to fit well. Both halves matter, and only the first one usually gets mentioned.\nWhat replaces it\nSampling. For each question GRPO samples a group of outputs, scores every one with a reward model — still present, still doing its job — and uses the group's average as the baseline. Those rewards get \"normalized by subtracting the group average and dividing by the group standard deviation\".\nSo an output's advantage answers a relative question: was this better or worse than its siblings? The paper argues that suits how reward models are built in the first place, since they're trained on comparisons between outputs for the same question.\nModel count goes four to three:\nPPO:\npolicy, reference, reward, value\nGRPO:\npolicy, reference, reward\nThe reward model is right there in the second list. It never left.\nThe trade that isn't in the headline\nNone of this is free. You now need several generations per question instead of one. DeepSeek-R1-Zero sampled 16 outputs per question.\nGRPO trades memory for generation. Memory-bound, that's a good deal. Throughput-bound, maybe not.\nThe KL term moved, it didn't vanish\nSecond thing I see stated wrong: that GRPO drops KL regularisation. PPO adds \"a per-token KL penalty from a reference model in the reward at each token\". GRPO works \"instead of adding KL penalty in the reward\" and \"regularizes by directly adding the KL divergence\" to the loss.\nSame idea, different address. GRPO also uses an unbiased estimator \"which is guaranteed to be positive\".\nThe finding I didn't expect\nThis one rarely survives into summaries, and it's the part I'd most want people to take away.\nThe DeepSeekMath authors measured Pass@K and Maj@K before and after RL. The result was asymmetric: \"RL enhances Maj@K's performance but not Pass@K\".\nRead that slowly. The model got better at reliably surfacing an answer it could already produce. It did not start producing answers that were previously out of reach. The authors draw the conclusion themselves — the gain is \"attributed to boosting the correct response from TopK rather than the enhancement of fundamental capabilities\".\nThat is a considerably narrower claim than the one usually made for RL fine-tuning.\nTwo things worth keeping straight\nThe value model went, the reward model stayed.\nIf someone says GRPO removed the reward model, they've merged two different networks.\nR1's rule-based rewards were a separate decision.\nDeepSeek-R1 did drop neural reward models for reasoning tasks, but for another reason entirely: \"neural reward models are susceptible to reward hacking during large-scale reinforcement learning\". That's a robustness call, not a memory one, and it applied to reasoning tasks specifically. It gets folded into the GRPO story where it doesn't belong.\nLonger version with every quote sourced at\ndiffstudy.com\n. Sources:\narXiv 2402.03300\n(DeepSeekMath),\narXiv 2501.12948\n(DeepSeek-R1),\narXiv 1707.06347\n(PPO).", "metadata": { "title": "GRPO doesn't remove the reward model. It removes the critic. - DEV Community", "description": "Every time GRPO comes up I see the same slip — someone says it \"gets rid of the reward model\". It... Tagged with ai, machinelearning, llm, computerscience.", "keywords": "ai, machinelearning, llm, computerscience, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/narotra05hp/grpo-doesnt-remove-the-reward-model-it-removes-the-critic-2pnp", "og:title": "GRPO doesn't remove the reward model. It removes the critic.", "og:description": "Every time GRPO comes up I see the same slip — someone says it \"gets rid of the reward model\". It...", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frmvvmdh407cfdh0u116s.png", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "GRPO doesn't remove the reward model. It removes the critic.", "twitter:description": "Every time GRPO comes up I see the same slip — someone says it \"gets rid of the reward model\". It...", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frmvvmdh407cfdh0u116s.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://diffstudy.com/grpo-vs-ppo/", "label": "diffstudy.com", "type": "a" }, { "href": "https://arxiv.org/abs/2402.03300", "label": "arXiv 2402.03300", "type": "a" }, { "href": "https://arxiv.org/abs/2501.12948", "label": "arXiv 2501.12948", "type": "a" }, { "href": "https://arxiv.org/abs/1707.06347", "label": "arXiv 1707.06347", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:49:00Z", "source_url": "https://dev.to/narotra05hp/grpo-doesnt-remove-the-reward-model-it-removes-the-critic-2pnp", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/2402.03300", "bytes": 1844642, "reused": true, "local_path": "papers/repair/dd0de0096b51ee9b7ed5.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2501.12948", "bytes": 5034998, "reused": true, "local_path": "papers/repair/d0ece4102f0cdb6b3ebb.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1707.06347", "bytes": 2923532, "reused": true, "local_path": "papers/repair/5c73fa759d96331836bf.pdf" } ], "actual_video_files": [], "body_chars": 4331, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/80ba6e9d1515/pages/source.html", "local_html": "attachments/80ba6e9d1515/pages/article.html", "body_base_path": "attachments/80ba6e9d1515/pages/", "body_markdown_path": "attachments/80ba6e9d1515/pages/article.md" }