{
  "source_file": "0930",
  "source_platform": "Ghost",
  "author": "Avi Chawla / Daily Dose of Data Science",
  "title_listed": "How To Beat GRPO Without Touching Model Weights",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.dailydoseofds.com/p/how-to-beat-grpo-without-touching-model-weights/",
  "publish_date": "2026-05-13",
  "corpus_stage": "",
  "notes": "",
  "id": "1458ea5d2311",
  "collected_at": "2026-10-02T13:48:12",
  "title": "How to Beat GRPO Without Touching Model Weights",
  "body_markdown": "[ Skip to main content  ](https://www.dailydoseofds.com/p/how-to-beat-grpo-without-touching-model-weights/#sx-main)\n[ ![Daily Dose of Data Science](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/12/dailyds-logo-transparent-1.png) ](https://www.dailydoseofds.com)\n  * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n  * [ Archive ](https://www.dailydoseofds.com/archive/)\n  * Guidebooks \n    * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n[ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n[ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n[ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n[ Courses ](https://www.dailydoseofds.com/)\n[ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n[ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n[ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n[ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n[ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n  * Courses \n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n[ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n[ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n[ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n[ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n[ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n[ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n[ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n  *     * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n[ Archive ](https://www.dailydoseofds.com/archive/)\nGuidebooks \n      * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n      * [ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n      * [ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n      * [ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n      * [ Courses ](https://www.dailydoseofds.com/)\n      * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n      * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n      * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n      * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n      * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\nCourses \n      * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n      * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n      * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n      * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n      * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n      * [ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n      * [ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n      * [ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n\n\nSign In  [ Get Started ](https://www.dailydoseofds.com/membership)\n/\n  * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n  * [ Archive ](https://www.dailydoseofds.com/archive/)\n  * Guidebooks \n    * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n    * [ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n    * [ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n    * [ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n    * [ Courses ](https://www.dailydoseofds.com/)\n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n    * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n    * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n    * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n    * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n  * Courses \n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n    * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n    * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n    * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n    * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n    * [ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n    * [ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n    * [ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n  * Sign In \n\n\nMay 2, 2026  Reinforcement Learning \n#  How to Beat GRPO Without Touching Model Weights \nBerkeley beat GRPO by 10 points with 35× fewer rollouts and no GPU training, \n![Avi Chawla](../images/000.jpg)\nAvi Chawla \n👉\n* * *\nGRPO needs tens of thousands of rollouts to converge. Each rollout produces a 5,000-token trace full of reasoning steps, tool calls, and self-corrections, but GRPO reduces all of it to a single scalar reward.\nSo we end up backpropagating on one bit per trajectory while throwing away thousands of bits of structured signal.\n![](../images/001.png)\nGEPA takes a different approach.\nInstead of computing policy gradients on that scalar, it hands the full rollout trace to a reflection LLM and asks “what went wrong, and how should the prompt change?”\nThe reflection model writes a new prompt, you test it, and if it improves, you keep it.\n![](../images/002.png)\nThe paper came out in July 2025. It was accepted at ICLR 2026, DSPy made it a first-class optimizer, and Hugging Face and OpenAI both shipped cookbooks around it.\nOn compound AI systems (multi-module pipelines with separate prompts), GEPA matches or beats GRPO while spending 10-50x less compute and requiring no training infrastructure at all.\nLet’s break down why it works, how it compares to GRPO, and how to use it in DSPy.\n> [**We started a course series on RL recently. Read Part 1 here →**](https://www.dailydoseofds.com/rl-course-part-1/)  \n>   \n>  This first chapter covers:what makes RL fundamentally different from supervised and unsupervised learningthe agent-environment interaction loopthe exploration-exploitation tradeoffmulti-armed bandits as the simplest RL setting, four action-selection strategies (greedy, ε-greedy, optimistic initialization, UCB)and a complete hands-on implementation of the classic 10-armed testbed with results and analysis.\n* * *\n#### **The signal compression problem in RL**\nReinforcement learning on language models has a signal problem that most practitioners overlook. Every rollout an agent produces is a 5,000-token document, containing:\n  * Reasoning steps\n  * Tool calls\n  * Self-corrections\n  * Compiler errors\n  * Judge rationales\n\n\nThat trace is rich and structured, containing exactly the kind of diagnostic information you’d want to learn from.\nWhile training the agent, GRPO takes all of that and reduces it to a single number.\n![](../images/003.png)\nAnd it throws away thousands of bits of structured info, which partly explains why it needs tens of thousands of rollouts to converge.\nThe signal isn’t sparse, but the final reward makes it sparse.\n#### **Letting the signal read itself**\nGEPA’s core idea is that the rollout is already a natural language artifact, so let an LLM read it. Don’t reduce the trace to a number.\nHand it to a reflection model along with the failure mode, and ask: “What went wrong here, and how should the prompt change?”\nThe reflection model writes a new prompt. You test it. And if it improves, you keep it.\n![](../images/004.png)\nThat’s the full optimization loop. Everything else in the paper is engineering that makes it work at scale.\n#### **What GEPA actually optimizes**\nGEPA targets compound AI systems.\nA pipeline of LLM modules with their own prompts, glued together by Python control flow. For instance, a multi-hop QA agent might have:\n  * A first-hop query writer\n  * A retriever\n  * A summarizer\n  * A second-hop query writer\n  * A final answerer\n\n![](../images/005.png)\nEach module has a prompt. GEPA evolves all of them.\nThe optimization target is simple: maximize expected metric on your task, subject to a rollout budget. The novelty is in how you spend that budget.\n#### **The feedback function**\nGEPA replaces your scalar metric with a feedback function μ_f.\nIt includes the same score that GRPO gives plus a natural language description of what happened.\n![](../images/006.png)\n  * For multi-hop QA, it returns which gold docs you retrieved and which you still need.\n  * For instruction-following, it returns per-constraint pass/fail descriptions.\n  * For code generation, it returns the actual compiler errors and profiler traces.\n  * For privacy-preserving rewriting, it splits the score into quality and PII-leakage with breakdowns.\n\n\n#### **The 6-step algorithm**\nEach iteration of the main loop does this:\n![](../images/007.png)\n  1. Pick a candidate prompt set from the population (Pareto sampling, more on this below)\n  2. Pick a module to mutate (round-robin across modules)\n  3. Sample 3 examples from the training set\n  4. Run rollouts and collect full traces plus feedback from μ_f\n  5. Reflect: feed traces and feedback to a reflection LLM, get a new prompt\n  6. Accept or reject: rerun on the same 3 examples. If better, keep it. If not, discard.\n\n\nRepeat until the budget runs out and return the best candidate. The entire loop runs without gradients, PPO, or KL penalties.\n#### **GEPA vs GRPO, head to head**\nA quick look at GRPO first:\n![](../images/008.gif)\nBoth GRPO and GEPA take feedback and improve the system. That’s where the similarity ends.\nGRPO updates model weights with policy gradients on scalar rewards. GEPA updates prompts with natural language reflection on full traces.\nHere’s the side-by-side comparison:\n![](../images/009.png)\nOne important caveat. GRPO can change what your model knows. GEPA can only change how you ask it.\nIf your base model can’t do the task at all, no prompt evolution will save you. Fine-tune when you need new capabilities. Use GEPA when you need to extract more from what’s already there.\n#### **A real example**\nLet’s make this concrete with a real example from the paper.\n**The task:** HotpotQA is a multi-hop question answering benchmark. You get a question that needs information from two different Wikipedia articles to answer. You can’t find the answer in just one place.\nExample question: “What is the population of the region containing the parish of São Vicente?”\nTo answer this, your agent has to:\n  1. 1st hop → Search “São Vicente parish”, retrieve a doc, learn it’s in Madeira.\n  2. 2nd hop → Search “Madeira population”, retrieve that doc, get the number.\n  3. Answer: Combine both\n\n\nThe agent has separate modules for each hop. We’re going to look at the prompt for the second-hop query writer, the module that decides what to search for after the first retrieval.\nInputs the module receives:\n  * question: the original user question\n  * summary_1: a summary of what the first hop retrieved\n\n\nOutput it produces:\n  * query: the search query for the second hop\n\n\nHere’s the seed prompt that DSPy gives you by default:\n> “Given the fields question, summary_1, produce the fields query.”\nThis is generic since it just describes the schema and it scores around 38% on validation.\nGEPA runs this on a few examples and watches what happens. For instance, the query writer might keep doing the same thing wrong, like it paraphrases the original question and retrieves the same documents it already had.\n![](../images/010.png)\nFor our São Vicente example, given a summary about the parish, it would search “São Vicente parish population” again, retrieve nothing new and fail.\nThe reflection LLM sees this failure pattern across multiple examples in the trace. It writes a new prompt:\n> “Generate a search query optimized for the second hop of multi-hop retrieval. The first-hop query was the original question, so first-hop docs already cover the entities mentioned directly. Your goal: retrieve documents NOT found in the first hop but necessary to answer completely. Avoid paraphrasing the original question. Target connected or higher-level entities mentioned in summary_1 but not explicitly in the question. Example: if summary_1 describes a parish but the question asks about the wider region’s total population, your query should target the region, not the parish. So for a question about São Vicente’s region, query ‘Madeira archipelago population’ rather than ‘São Vicente population’.”\nThat rewritten prompt scores 69%, up from 38% on the seed.\nThe model and task stayed identical. The only thing that changed was the prompt for one module out of several.\nThe reflection LLM didn’t just rephrase the seed but it also absorbed an actual strategy from observed failures:\n  * Don’t paraphrase the question\n  * The first hop already covered the directly-mentioned entities\n  * Target the broader entity that connects them\n  * Here’s a worked example of the pattern\n\n\nThat’s the kind of information you cannot encode in a policy gradient. RL would just tell you “this trajectory was 0.3 below the group mean” and let backprop figure it out across thousands of tokens. GEPA writes the lesson down in plain English and ships it as the new prompt.\n#### **Pareto selection**\nThe idea of evolving/optimizing the prompt is not new.\nBut most approaches often mutate from the best candidate so far, which sounds reasonable, but it collapses to local optima fast.\nGEPA uses something smarter, borrowed from quality-diversity optimization.\nImagine three candidate prompts and four tasks:\n![](../images/011.png)\nA greedy approach will pick C every time since it has the best average.\nBut A is the only one who handles Task 1 well, and B does Task 2 well. If you only mutate C, you lose those strategies forever.\nPareto selection keeps anyone who’s best at at least one task. Then it samples parents weighted by how many tasks they win. So C is most likely to be picked, but A and B stay in the pool. Their distinctive strengths can later be combined with C’s.\n![](../images/012.png)\nThis single design choice is what separates GEPA from earlier evolutionary prompt methods.\n#### **Where GEPA fits in the landscape**\nQuick map of who does what:\n![](../images/013.png)\n  * **APE, OPRO:** Both use an LLM to propose prompt candidates scored by a scalar metric. APE generates candidates from input-output demos and picks the best. OPRO feeds prior prompts and their scores into a meta-prompt so the LLM can propose informed improvements. Single prompt, no reflection on traces.\n  * **EvoPrompt, Promptbreeder:** Evolutionary operators (crossover, mutation) applied to prompt populations via LLM calls. Promptbreeder adds a self-referential layer: it also evolves the mutation-prompts themselves. Both use scalar fitness for selection and target single prompts, not multi-module pipelines.\n  * **Reflexion:** Agents reflect on task feedback after each trial and store reflections in an episodic memory buffer for the next attempt. Improves per-instance behavior across retries, not population-level prompt evolution across a training set.\n  * **TextGrad:** PyTorch-style backpropagation but with natural language critiques instead of numerical gradients. An LLM propagates textual feedback through a computation graph to produce per-variable improvement suggestions. Single candidate per iteration, no population.\n  * **MIPROv2:** DSPy’s prior flagship. Bootstraps few-shot examples from training data, proposes instruction candidates grounded in data summaries and traces, then uses Bayesian Optimization (Optuna TPE) to search the joint instruction-demo space across all modules. Generates all candidates upfront rather than evolving them through reflection.\n  * **GRPO:** Actual RL with weight updates. Samples a group of rollouts per prompt, uses the group mean as baseline to compute per-trajectory advantage, then updates weights via policy gradients with a KL penalty. The only method here that changes what the model knows, not just how you prompt it.\n\n\nGEPA borrows ideas from several of these methods. It uses verbal reflection like Reflexion, but applies it across a population of candidates instead of a single agent’s memory.\nIt evolves that population using selection pressure like EvoPrompt, but with natural language feedback driving mutations instead of scalar fitness.\nIt targets compound multi-module pipelines like MIPROv2, but evolves prompts iteratively through reflection instead of generating all candidates upfront.\nThe piece that’s new to GEPA is Pareto selection, which preserves candidates that are best at even one task rather than always mutating from the highest-average performer.\n#### **Using it in DSPy**\nThe API is one line different from MIPROv2:\n\n```\noptimizer = dspy.GEPA(\n    metric=metric_with_feedback,\n    auto=\"medium\",\n    reflection_minibatch_size=3,\n    candidate_selection_strategy=\"pareto\",\n    reflection_lm=dspy.LM(\"gpt-5\", temperature=1.0, max_tokens=32000),\n    use_merge=True,\n    track_stats=True,\n)\n\noptimized = optimizer.compile(program, trainset=train, valset=val)\n```\n\nCopy\nThe catch: your metric function needs the right signature.\n\n```\ndef metric(gold, pred, trace=None, pred_name=None, pred_trace=None):\n    # return dspy.Prediction(score=float, feedback=str)\n```\n\nCopy\nReturn a prediction with both a score and a feedback string. The feedback is what gets fed to the reflection LLM, so make it diagnostic and specific.\nIf your feedback string is just “wrong answer”, you’re back to scalar territory, and GEPA degrades to a slower MIPROv2.\nIf your feedback says “missed entity X, retrieved doc Y when gold was Z, format violation in step 3”, then GEPA works well with that level of detail.\n#### **A 2026 reality check**\nGEPA beats GRPO specifically, not every RL method.\nThe field has stopped framing this as GEPA vs RL and started framing it as GEPA and RL. The paper itself points to hybrid recipes as the natural next step.\nReflection is far more sample-efficient than RL on compound systems. The two are increasingly combined, not pitted against each other.\nOne more nuance worth knowing is that Decagon’s March 2026 production ablations found that more data isn’t always better with GEPA. 20 to 100 examples often beats 500. The reflection loop overfits when you feed it too much.\nGEPA learns from patterns in failures. With 50 well-chosen examples, the reflector sees a clean signal. With 500, it starts chasing noise.\nUse small, high-quality training sets and don’t assume scale helps.\n#### **When to reach for what**\nIf you’re building a compound AI system today, here’s the decision tree.\n![](../images/014.png)\n  * **Use GEPA when:** You have a small training set, expensive rollouts, no access to weights, and a metric you can describe in words.\n  * **Use GRPO when:** You have abundant cheap rollouts, open weights, and a verifiable terminal reward.\n  * **Use MIPROv2 when:** You specifically need bootstrapped few-shot exemplars in your prompts.\n  * **Use TextGrad when:** Your computation graph is deep and you want explicit per-variable critique propagation.\n\n\nFor most practical compound-system work in 2026, GEPA is the default to try first.\nRL still has its place, but it’s no longer the obvious default when reading a rollout costs less than running ten thousand more.\n**To dive deeper into RL,**[**we started a course series on RL recently. Read Part 1 here →**](https://www.dailydoseofds.com/rl-course-part-1/)\nThe first chapter covers:\n  * what makes RL fundamentally different from supervised and unsupervised learning\n  * the agent-environment interaction loop\n  * the exploration-exploitation tradeoff\n  * multi-armed bandits as the simplest RL setting, four action-selection strategies (greedy, ε-greedy, optimistic initialization, UCB)\n  * and a complete hands-on implementation of the classic 10-armed testbed with results and analysis.\n\n\nThanks for reading!\nPublished on May 2, 2026 \nComments \nShare \nCopy link [ Share to X ](https://x.com/intent/tweet?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-to-beat-grpo-without-touching-model-weights%2F&text=How%20to%20Beat%20GRPO%20Without%20Touching%20Model%20Weights) [ Share to Facebook ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-to-beat-grpo-without-touching-model-weights%2F) [ Share to Linkedin ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-to-beat-grpo-without-touching-model-weights%2F)\nCopied \n[ Previous  How Top AI Labs Are Building RL Agents in 2026  ](https://www.dailydoseofds.com/p/how-top-ai-labs-are-building-rl-agents-in-2026/) [ Next  Markov Decision Processes and Value Functions  ](https://www.dailydoseofds.com/rl-course-part-2/)\n[ ![Daily Dose of Data Science](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/12/dailyds-logo-transparent-1.png) ](https://www.dailydoseofds.com)\nA daily column with insights, observations, tutorials and best practices on python and data science. Read by industry professionals at big tech, startups, and engineering students. \n  * Menu \n  * [ Contact ](https://www.dailydoseofds.com/contact/)\n  * [ FAQ ](https://www.dailydoseofds.com/faq/)\n\n\n[Daily Dose of Data Science](https://www.dailydoseofds.com) © 2026 \n[ ](https://x.com/_avichawla \"X \\(Formerly Twitter\\)\")\n",
  "metadata": {
    "title": "How to Beat GRPO Without Touching Model Weights",
    "description": "Berkeley beat GRPO by 10 points with 35× fewer rollouts and no GPU training,",
    "keywords": null,
    "author": null,
    "og:site_name": "Daily Dose of Data Science",
    "og:type": "article",
    "og:title": "How to Beat GRPO Without Touching Model Weights",
    "og:description": "Berkeley beat GRPO by 10 points with 35× fewer rollouts and no GPU training,",
    "og:url": "https://www.dailydoseofds.com/p/how-to-beat-grpo-without-touching-model-weights/",
    "og:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/05/clipboard-image-1778702710.png",
    "og:image:width": "680",
    "og:image:height": "673",
    "twitter:card": "summary_large_image",
    "twitter:title": "How to Beat GRPO Without Touching Model Weights",
    "twitter:description": "Berkeley beat GRPO by 10 points with 35× fewer rollouts and no GPU training,",
    "twitter:url": "https://www.dailydoseofds.com/p/how-to-beat-grpo-without-touching-model-weights/",
    "twitter:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/05/clipboard-image-1778702710.png",
    "twitter:label1": "Written by",
    "twitter:data1": "Avi Chawla",
    "twitter:label2": "Filed under",
    "twitter:data2": "Reinforcement Learning",
    "twitter:site": "@_avichawla",
    "article:published_time": "2026-05-01T20:08:00.000Z",
    "article:modified_time": "2026-05-13T20:08:45.000Z",
    "article:tag": "Reinforcement Learning"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/1458ea5d2311/code/block_000.txt",
      "chars": 334
    },
    {
      "language": "txt",
      "file": "attachments/1458ea5d2311/code/block_001.txt",
      "chars": 125
    }
  ],
  "images": [
    {
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "alt": "Avi Chawla",
      "local_path": "attachments/1458ea5d2311/images/000.jpg",
      "bytes": 11344,
      "downloaded": true,
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "reused": true,
      "retried_at": "2026-10-18T17:32:11Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!oLVq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F565d9d1f-4514-47be-8a56-bcff6ddbd556_1274x642.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/001.png",
      "bytes": 332944,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!oLVq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F565d9d1f-4514-47be-8a56-bcff6ddbd556_1274x642.png",
      "reused": true,
      "retried_at": "2026-10-18T17:32:11Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!BqUJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cf73c7b-cc25-4a19-b783-6e8c6ada48bc_851x455.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/002.png",
      "bytes": 174789,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!BqUJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cf73c7b-cc25-4a19-b783-6e8c6ada48bc_851x455.png",
      "reused": true,
      "retried_at": "2026-10-18T17:32:11Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!WAG7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F816d7953-f8d3-480a-84d5-e38f3f299e42_680x378.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/003.png",
      "bytes": 100034,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!WAG7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F816d7953-f8d3-480a-84d5-e38f3f299e42_680x378.png",
      "reused": true,
      "retried_at": "2026-10-18T17:32:11Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!HD_5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e61481b-0911-40ff-8d0e-e74bfb67842b_679x358.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/004.png",
      "bytes": 120077,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!HD_5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e61481b-0911-40ff-8d0e-e74bfb67842b_679x358.png",
      "reused": true,
      "retried_at": "2026-10-18T17:32:11Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!IN25!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2947deb4-7e47-4ead-a2f3-d1f9d6cfe724_680x381.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/005.png",
      "bytes": 105891,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!IN25!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2947deb4-7e47-4ead-a2f3-d1f9d6cfe724_680x381.png",
      "reused": true,
      "retried_at": "2026-10-18T17:32:11Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!h7Xk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6a7c5c0-3243-459d-bfd9-0ce59be84fc1_680x379.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/006.png",
      "bytes": 135870,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!h7Xk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6a7c5c0-3243-459d-bfd9-0ce59be84fc1_680x379.png",
      "reused": true,
      "retried_at": "2026-10-18T17:32:12Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!lzNT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e3d9fa5-9761-4417-970d-371bf6dfaeea_680x418.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/007.png",
      "bytes": 106602,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!lzNT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e3d9fa5-9761-4417-970d-371bf6dfaeea_680x418.png",
      "reused": true,
      "retried_at": "2026-10-18T17:32:12Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!TefD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb14147e7-8eea-4b15-a073-38acba8eeb23_1080x1029.gif",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/008.gif",
      "bytes": 625265,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!TefD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb14147e7-8eea-4b15-a073-38acba8eeb23_1080x1029.gif",
      "reused": true,
      "retried_at": "2026-10-18T17:33:13Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!HDmM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ceaf7f5-2ab4-4025-b691-486ae7692802_680x377.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/009.png",
      "bytes": 48941,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!HDmM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ceaf7f5-2ab4-4025-b691-486ae7692802_680x377.png",
      "reused": true,
      "retried_at": "2026-10-18T17:33:13Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!YEs2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c88fc2-5a78-4bbc-a5cf-c47f0cb427ed_680x375.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/010.png",
      "bytes": 122783,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!YEs2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c88fc2-5a78-4bbc-a5cf-c47f0cb427ed_680x375.png",
      "reused": true,
      "retried_at": "2026-10-18T17:33:13Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!72IS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F170f1ea5-3d56-45e8-bcdf-96064bd10745_679x235.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/011.png",
      "bytes": 30640,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!72IS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F170f1ea5-3d56-45e8-bcdf-96064bd10745_679x235.png",
      "reused": true,
      "retried_at": "2026-10-18T17:33:13Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!u8qJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a800d5b-6af3-4559-811c-d106c2ba96e1_680x379.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/012.png",
      "bytes": 111631,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!u8qJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a800d5b-6af3-4559-811c-d106c2ba96e1_680x379.png",
      "reused": true,
      "retried_at": "2026-10-18T17:33:13Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!NS4I!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff44c3776-25c7-4a10-987b-534e87996891_680x379.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/013.png",
      "bytes": 99508,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!NS4I!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff44c3776-25c7-4a10-987b-534e87996891_680x379.png",
      "reused": true,
      "retried_at": "2026-10-18T17:33:13Z"
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!t1W9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe997d92d-2850-4831-a58e-00edb4ae1b19_680x377.png",
      "alt": "",
      "local_path": "attachments/1458ea5d2311/images/014.png",
      "bytes": 119421,
      "downloaded": true,
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!t1W9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe997d92d-2850-4831-a58e-00edb4ae1b19_680x377.png",
      "reused": true,
      "retried_at": "2026-10-18T17:33:13Z"
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/p/how-to-beat-grpo-without-touching-model-weights/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/archive/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/what-is-mcp/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/mlops-crash-course-part-1/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/rl-course-part-1/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/membership"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/p/how-top-ai-labs-are-building-rl-agents-in-2026/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/rl-course-part-2/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/contact/"
    },
    {
      "type": "internal",
      "href": "https://www.dailydoseofds.com/faq/"
    },
    {
      "type": "external",
      "href": "https://dailydoseofds.github.io/ai-engg-book/"
    },
    {
      "type": "external",
      "href": "https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view"
    },
    {
      "type": "external",
      "href": "https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view"
    },
    {
      "type": "external",
      "href": "https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view"
    },
    {
      "type": "external",
      "href": "https://x.com/intent/tweet?text=How+to+Beat+GRPO+Without+Touching+Model+Weights&url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-to-beat-grpo-without-touching-model-weights%2F"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-to-beat-grpo-without-touching-model-weights%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-to-beat-grpo-without-touching-model-weights%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/_avichawla"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:33:17Z",
  "source_url": "https://www.dailydoseofds.com/p/how-to-beat-grpo-without-touching-model-weights/",
  "delivery_status": "unverified",
  "discovery_status": "short_or_dynamic",
  "missing_reason": "正文过短或动态页面，未确认完整",
  "source_body_markdown": "[ Skip to main content  ](https://www.dailydoseofds.com/p/how-to-beat-grpo-without-touching-model-weights/#sx-main)\n[ ![Daily Dose of Data Science](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/12/dailyds-logo-transparent-1.png) ](https://www.dailydoseofds.com)\n  * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n  * [ Archive ](https://www.dailydoseofds.com/archive/)\n  * Guidebooks \n    * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n[ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n[ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n[ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n[ Courses ](https://www.dailydoseofds.com/)\n[ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n[ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n[ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n[ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n[ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n  * Courses \n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n[ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n[ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n[ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n[ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n[ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n[ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n[ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n  *     * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n[ Archive ](https://www.dailydoseofds.com/archive/)\nGuidebooks \n      * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n      * [ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n      * [ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n      * [ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n      * [ Courses ](https://www.dailydoseofds.com/)\n      * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n      * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n      * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n      * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n      * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\nCourses \n      * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n      * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n      * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n      * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n      * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n      * [ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n      * [ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n      * [ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n\n\nSign In  [ Get Started ](https://www.dailydoseofds.com/membership)\n/\n  * [ Newsletter ](https://dailydoseofds.github.io/ai-engg-book/)\n  * [ Archive ](https://www.dailydoseofds.com/archive/)\n  * Guidebooks \n    * [ MCP Guidebook ](https://www.dailydoseofds.com/what-is-mcp/)\n    * [ Agents Guidebook ](https://drive.google.com/file/d/1Th8mN_IF7Ttc8bI_OLtUuQ7Mjx3aJ6Hi/view)\n    * [ AI Engineering Guidebook ](https://drive.google.com/file/d/1_s2UaD7tEPECjgC5YMOa1LIOB9HUWFO-/view)\n    * [ DS Guidebook ](https://drive.google.com/file/d/1ZunUHwkD2Ut0_LrgDTDmlQRQkfT2lkVf/view)\n    * [ Courses ](https://www.dailydoseofds.com/)\n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n    * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n    * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n    * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n    * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n  * Courses \n    * [ MLOps/LLMOps Course ](https://www.dailydoseofds.com/mlops-crash-course-part-1/)\n    * [ Reinforcement Learning Course ](https://www.dailydoseofds.com/rl-course-part-1/)\n    * [ AI Agents Course ](https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/)\n    * [ MCP Course ](https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/)\n    * [ RAG Systems Course ](https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/)\n    * [ LLM and Fine-tuning ](https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/)\n    * [ Classical ML and Deep Learning ](https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/)\n    * [ Engineering Best Practices ](https://www.dailydoseofds.com/8-fatal-yet-non-obvious-pitfalls-and-cautionary-measures-in-data-science/)\n  * Sign In \n\n\nMay 2, 2026  Reinforcement Learning \n#  How to Beat GRPO Without Touching Model Weights \nBerkeley beat GRPO by 10 points with 35× fewer rollouts and no GPU training, \n![Avi Chawla](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg)\nAvi Chawla \n👉\n* * *\nGRPO needs tens of thousands of rollouts to converge. Each rollout produces a 5,000-token trace full of reasoning steps, tool calls, and self-corrections, but GRPO reduces all of it to a single scalar reward.\nSo we end up backpropagating on one bit per trajectory while throwing away thousands of bits of structured signal.\n![](https://substackcdn.com/image/fetch/$s_!oLVq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F565d9d1f-4514-47be-8a56-bcff6ddbd556_1274x642.png)\nGEPA takes a different approach.\nInstead of computing policy gradients on that scalar, it hands the full rollout trace to a reflection LLM and asks “what went wrong, and how should the prompt change?”\nThe reflection model writes a new prompt, you test it, and if it improves, you keep it.\n![](https://substackcdn.com/image/fetch/$s_!BqUJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cf73c7b-cc25-4a19-b783-6e8c6ada48bc_851x455.png)\nThe paper came out in July 2025. It was accepted at ICLR 2026, DSPy made it a first-class optimizer, and Hugging Face and OpenAI both shipped cookbooks around it.\nOn compound AI systems (multi-module pipelines with separate prompts), GEPA matches or beats GRPO while spending 10-50x less compute and requiring no training infrastructure at all.\nLet’s break down why it works, how it compares to GRPO, and how to use it in DSPy.\n> [**We started a course series on RL recently. Read Part 1 here →**](https://www.dailydoseofds.com/rl-course-part-1/)  \n>   \n>  This first chapter covers:what makes RL fundamentally different from supervised and unsupervised learningthe agent-environment interaction loopthe exploration-exploitation tradeoffmulti-armed bandits as the simplest RL setting, four action-selection strategies (greedy, ε-greedy, optimistic initialization, UCB)and a complete hands-on implementation of the classic 10-armed testbed with results and analysis.\n* * *\n#### **The signal compression problem in RL**\nReinforcement learning on language models has a signal problem that most practitioners overlook. Every rollout an agent produces is a 5,000-token document, containing:\n  * Reasoning steps\n  * Tool calls\n  * Self-corrections\n  * Compiler errors\n  * Judge rationales\n\n\nThat trace is rich and structured, containing exactly the kind of diagnostic information you’d want to learn from.\nWhile training the agent, GRPO takes all of that and reduces it to a single number.\n![](https://substackcdn.com/image/fetch/$s_!WAG7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F816d7953-f8d3-480a-84d5-e38f3f299e42_680x378.png)\nAnd it throws away thousands of bits of structured info, which partly explains why it needs tens of thousands of rollouts to converge.\nThe signal isn’t sparse, but the final reward makes it sparse.\n#### **Letting the signal read itself**\nGEPA’s core idea is that the rollout is already a natural language artifact, so let an LLM read it. Don’t reduce the trace to a number.\nHand it to a reflection model along with the failure mode, and ask: “What went wrong here, and how should the prompt change?”\nThe reflection model writes a new prompt. You test it. And if it improves, you keep it.\n![](https://substackcdn.com/image/fetch/$s_!HD_5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e61481b-0911-40ff-8d0e-e74bfb67842b_679x358.png)\nThat’s the full optimization loop. Everything else in the paper is engineering that makes it work at scale.\n#### **What GEPA actually optimizes**\nGEPA targets compound AI systems.\nA pipeline of LLM modules with their own prompts, glued together by Python control flow. For instance, a multi-hop QA agent might have:\n  * A first-hop query writer\n  * A retriever\n  * A summarizer\n  * A second-hop query writer\n  * A final answerer\n\n![](https://substackcdn.com/image/fetch/$s_!IN25!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2947deb4-7e47-4ead-a2f3-d1f9d6cfe724_680x381.png)\nEach module has a prompt. GEPA evolves all of them.\nThe optimization target is simple: maximize expected metric on your task, subject to a rollout budget. The novelty is in how you spend that budget.\n#### **The feedback function**\nGEPA replaces your scalar metric with a feedback function μ_f.\nIt includes the same score that GRPO gives plus a natural language description of what happened.\n![](https://substackcdn.com/image/fetch/$s_!h7Xk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6a7c5c0-3243-459d-bfd9-0ce59be84fc1_680x379.png)\n  * For multi-hop QA, it returns which gold docs you retrieved and which you still need.\n  * For instruction-following, it returns per-constraint pass/fail descriptions.\n  * For code generation, it returns the actual compiler errors and profiler traces.\n  * For privacy-preserving rewriting, it splits the score into quality and PII-leakage with breakdowns.\n\n\n#### **The 6-step algorithm**\nEach iteration of the main loop does this:\n![](https://substackcdn.com/image/fetch/$s_!lzNT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e3d9fa5-9761-4417-970d-371bf6dfaeea_680x418.png)\n  1. Pick a candidate prompt set from the population (Pareto sampling, more on this below)\n  2. Pick a module to mutate (round-robin across modules)\n  3. Sample 3 examples from the training set\n  4. Run rollouts and collect full traces plus feedback from μ_f\n  5. Reflect: feed traces and feedback to a reflection LLM, get a new prompt\n  6. Accept or reject: rerun on the same 3 examples. If better, keep it. If not, discard.\n\n\nRepeat until the budget runs out and return the best candidate. The entire loop runs without gradients, PPO, or KL penalties.\n#### **GEPA vs GRPO, head to head**\nA quick look at GRPO first:\n![](https://substackcdn.com/image/fetch/$s_!TefD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb14147e7-8eea-4b15-a073-38acba8eeb23_1080x1029.gif)\nBoth GRPO and GEPA take feedback and improve the system. That’s where the similarity ends.\nGRPO updates model weights with policy gradients on scalar rewards. GEPA updates prompts with natural language reflection on full traces.\nHere’s the side-by-side comparison:\n![](https://substackcdn.com/image/fetch/$s_!HDmM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ceaf7f5-2ab4-4025-b691-486ae7692802_680x377.png)\nOne important caveat. GRPO can change what your model knows. GEPA can only change how you ask it.\nIf your base model can’t do the task at all, no prompt evolution will save you. Fine-tune when you need new capabilities. Use GEPA when you need to extract more from what’s already there.\n#### **A real example**\nLet’s make this concrete with a real example from the paper.\n**The task:** HotpotQA is a multi-hop question answering benchmark. You get a question that needs information from two different Wikipedia articles to answer. You can’t find the answer in just one place.\nExample question: “What is the population of the region containing the parish of São Vicente?”\nTo answer this, your agent has to:\n  1. 1st hop → Search “São Vicente parish”, retrieve a doc, learn it’s in Madeira.\n  2. 2nd hop → Search “Madeira population”, retrieve that doc, get the number.\n  3. Answer: Combine both\n\n\nThe agent has separate modules for each hop. We’re going to look at the prompt for the second-hop query writer, the module that decides what to search for after the first retrieval.\nInputs the module receives:\n  * question: the original user question\n  * summary_1: a summary of what the first hop retrieved\n\n\nOutput it produces:\n  * query: the search query for the second hop\n\n\nHere’s the seed prompt that DSPy gives you by default:\n> “Given the fields question, summary_1, produce the fields query.”\nThis is generic since it just describes the schema and it scores around 38% on validation.\nGEPA runs this on a few examples and watches what happens. For instance, the query writer might keep doing the same thing wrong, like it paraphrases the original question and retrieves the same documents it already had.\n![](https://substackcdn.com/image/fetch/$s_!YEs2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c88fc2-5a78-4bbc-a5cf-c47f0cb427ed_680x375.png)\nFor our São Vicente example, given a summary about the parish, it would search “São Vicente parish population” again, retrieve nothing new and fail.\nThe reflection LLM sees this failure pattern across multiple examples in the trace. It writes a new prompt:\n> “Generate a search query optimized for the second hop of multi-hop retrieval. The first-hop query was the original question, so first-hop docs already cover the entities mentioned directly. Your goal: retrieve documents NOT found in the first hop but necessary to answer completely. Avoid paraphrasing the original question. Target connected or higher-level entities mentioned in summary_1 but not explicitly in the question. Example: if summary_1 describes a parish but the question asks about the wider region’s total population, your query should target the region, not the parish. So for a question about São Vicente’s region, query ‘Madeira archipelago population’ rather than ‘São Vicente population’.”\nThat rewritten prompt scores 69%, up from 38% on the seed.\nThe model and task stayed identical. The only thing that changed was the prompt for one module out of several.\nThe reflection LLM didn’t just rephrase the seed but it also absorbed an actual strategy from observed failures:\n  * Don’t paraphrase the question\n  * The first hop already covered the directly-mentioned entities\n  * Target the broader entity that connects them\n  * Here’s a worked example of the pattern\n\n\nThat’s the kind of information you cannot encode in a policy gradient. RL would just tell you “this trajectory was 0.3 below the group mean” and let backprop figure it out across thousands of tokens. GEPA writes the lesson down in plain English and ships it as the new prompt.\n#### **Pareto selection**\nThe idea of evolving/optimizing the prompt is not new.\nBut most approaches often mutate from the best candidate so far, which sounds reasonable, but it collapses to local optima fast.\nGEPA uses something smarter, borrowed from quality-diversity optimization.\nImagine three candidate prompts and four tasks:\n![](https://substackcdn.com/image/fetch/$s_!72IS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F170f1ea5-3d56-45e8-bcdf-96064bd10745_679x235.png)\nA greedy approach will pick C every time since it has the best average.\nBut A is the only one who handles Task 1 well, and B does Task 2 well. If you only mutate C, you lose those strategies forever.\nPareto selection keeps anyone who’s best at at least one task. Then it samples parents weighted by how many tasks they win. So C is most likely to be picked, but A and B stay in the pool. Their distinctive strengths can later be combined with C’s.\n![](https://substackcdn.com/image/fetch/$s_!u8qJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a800d5b-6af3-4559-811c-d106c2ba96e1_680x379.png)\nThis single design choice is what separates GEPA from earlier evolutionary prompt methods.\n#### **Where GEPA fits in the landscape**\nQuick map of who does what:\n![](https://substackcdn.com/image/fetch/$s_!NS4I!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff44c3776-25c7-4a10-987b-534e87996891_680x379.png)\n  * **APE, OPRO:** Both use an LLM to propose prompt candidates scored by a scalar metric. APE generates candidates from input-output demos and picks the best. OPRO feeds prior prompts and their scores into a meta-prompt so the LLM can propose informed improvements. Single prompt, no reflection on traces.\n  * **EvoPrompt, Promptbreeder:** Evolutionary operators (crossover, mutation) applied to prompt populations via LLM calls. Promptbreeder adds a self-referential layer: it also evolves the mutation-prompts themselves. Both use scalar fitness for selection and target single prompts, not multi-module pipelines.\n  * **Reflexion:** Agents reflect on task feedback after each trial and store reflections in an episodic memory buffer for the next attempt. Improves per-instance behavior across retries, not population-level prompt evolution across a training set.\n  * **TextGrad:** PyTorch-style backpropagation but with natural language critiques instead of numerical gradients. An LLM propagates textual feedback through a computation graph to produce per-variable improvement suggestions. Single candidate per iteration, no population.\n  * **MIPROv2:** DSPy’s prior flagship. Bootstraps few-shot examples from training data, proposes instruction candidates grounded in data summaries and traces, then uses Bayesian Optimization (Optuna TPE) to search the joint instruction-demo space across all modules. Generates all candidates upfront rather than evolving them through reflection.\n  * **GRPO:** Actual RL with weight updates. Samples a group of rollouts per prompt, uses the group mean as baseline to compute per-trajectory advantage, then updates weights via policy gradients with a KL penalty. The only method here that changes what the model knows, not just how you prompt it.\n\n\nGEPA borrows ideas from several of these methods. It uses verbal reflection like Reflexion, but applies it across a population of candidates instead of a single agent’s memory.\nIt evolves that population using selection pressure like EvoPrompt, but with natural language feedback driving mutations instead of scalar fitness.\nIt targets compound multi-module pipelines like MIPROv2, but evolves prompts iteratively through reflection instead of generating all candidates upfront.\nThe piece that’s new to GEPA is Pareto selection, which preserves candidates that are best at even one task rather than always mutating from the highest-average performer.\n#### **Using it in DSPy**\nThe API is one line different from MIPROv2:\n\n```\noptimizer = dspy.GEPA(\n    metric=metric_with_feedback,\n    auto=\"medium\",\n    reflection_minibatch_size=3,\n    candidate_selection_strategy=\"pareto\",\n    reflection_lm=dspy.LM(\"gpt-5\", temperature=1.0, max_tokens=32000),\n    use_merge=True,\n    track_stats=True,\n)\n\noptimized = optimizer.compile(program, trainset=train, valset=val)\n```\n\nCopy\nThe catch: your metric function needs the right signature.\n\n```\ndef metric(gold, pred, trace=None, pred_name=None, pred_trace=None):\n    # return dspy.Prediction(score=float, feedback=str)\n```\n\nCopy\nReturn a prediction with both a score and a feedback string. The feedback is what gets fed to the reflection LLM, so make it diagnostic and specific.\nIf your feedback string is just “wrong answer”, you’re back to scalar territory, and GEPA degrades to a slower MIPROv2.\nIf your feedback says “missed entity X, retrieved doc Y when gold was Z, format violation in step 3”, then GEPA works well with that level of detail.\n#### **A 2026 reality check**\nGEPA beats GRPO specifically, not every RL method.\nThe field has stopped framing this as GEPA vs RL and started framing it as GEPA and RL. The paper itself points to hybrid recipes as the natural next step.\nReflection is far more sample-efficient than RL on compound systems. The two are increasingly combined, not pitted against each other.\nOne more nuance worth knowing is that Decagon’s March 2026 production ablations found that more data isn’t always better with GEPA. 20 to 100 examples often beats 500. The reflection loop overfits when you feed it too much.\nGEPA learns from patterns in failures. With 50 well-chosen examples, the reflector sees a clean signal. With 500, it starts chasing noise.\nUse small, high-quality training sets and don’t assume scale helps.\n#### **When to reach for what**\nIf you’re building a compound AI system today, here’s the decision tree.\n![](https://substackcdn.com/image/fetch/$s_!t1W9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe997d92d-2850-4831-a58e-00edb4ae1b19_680x377.png)\n  * **Use GEPA when:** You have a small training set, expensive rollouts, no access to weights, and a metric you can describe in words.\n  * **Use GRPO when:** You have abundant cheap rollouts, open weights, and a verifiable terminal reward.\n  * **Use MIPROv2 when:** You specifically need bootstrapped few-shot exemplars in your prompts.\n  * **Use TextGrad when:** Your computation graph is deep and you want explicit per-variable critique propagation.\n\n\nFor most practical compound-system work in 2026, GEPA is the default to try first.\nRL still has its place, but it’s no longer the obvious default when reading a rollout costs less than running ten thousand more.\n**To dive deeper into RL,**[**we started a course series on RL recently. Read Part 1 here →**](https://www.dailydoseofds.com/rl-course-part-1/)\nThe first chapter covers:\n  * what makes RL fundamentally different from supervised and unsupervised learning\n  * the agent-environment interaction loop\n  * the exploration-exploitation tradeoff\n  * multi-armed bandits as the simplest RL setting, four action-selection strategies (greedy, ε-greedy, optimistic initialization, UCB)\n  * and a complete hands-on implementation of the classic 10-armed testbed with results and analysis.\n\n\nThanks for reading!\nPublished on May 2, 2026 \nComments \nShare \nCopy link [ Share to X ](https://x.com/intent/tweet?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-to-beat-grpo-without-touching-model-weights%2F&text=How%20to%20Beat%20GRPO%20Without%20Touching%20Model%20Weights) [ Share to Facebook ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-to-beat-grpo-without-touching-model-weights%2F) [ Share to Linkedin ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-to-beat-grpo-without-touching-model-weights%2F)\nCopied \n[ Previous  How Top AI Labs Are Building RL Agents in 2026  ](https://www.dailydoseofds.com/p/how-top-ai-labs-are-building-rl-agents-in-2026/) [ Next  Markov Decision Processes and Value Functions  ](https://www.dailydoseofds.com/rl-course-part-2/)\n[ ![Daily Dose of Data Science](https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2025/12/dailyds-logo-transparent-1.png) ](https://www.dailydoseofds.com)\nA daily column with insights, observations, tutorials and best practices on python and data science. Read by industry professionals at big tech, startups, and engineering students. \n  * Menu \n  * [ Contact ](https://www.dailydoseofds.com/contact/)\n  * [ FAQ ](https://www.dailydoseofds.com/faq/)\n\n\n[Daily Dose of Data Science](https://www.dailydoseofds.com) © 2026 \n[ ](https://x.com/_avichawla \"X \\(Formerly Twitter\\)\")\n",
  "body_base_path": "attachments/1458ea5d2311/pages/",
  "body_markdown_path": "attachments/1458ea5d2311/pages/archived-body.md",
  "local_html": "attachments/1458ea5d2311/pages/archived-body.html",
  "body_source": "prior_capture"
}