{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "AI by Hand ✍️\nProf. Tom Yeh",
  "title_listed": "PPO → DPO → GRPO → Rubrics: The Evolution of RL for LLMs",
  "keywords": "PPO, DPO, GRPO, 强化学习, RLHF, 对齐训练",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "付费",
  "url": "https://www.byhand.ai/p/recording-ppo-dpo-grpo-rubrics",
  "publish_date": "2026-03",
  "corpus_stage": "RLHF/对齐",
  "notes": "系统梳理LLM强化学习方法演进：从PPO到DPO再到GRPO的完整技术路线",
  "id": "46f4fdd906da",
  "collected_at": "2026-10-01T20:20:48",
  "title": "PPO → DPO → GRPO → Rubrics (Mar 2, 2026) - by Prof. Tom Yeh",
  "body_markdown": "[![AI by Hand ✍️](https://substackcdn.com/image/fetch/$s_!fOmN!,w_40,h_40,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F618d59fa-9b8f-4814-8d16-f29c2f23ae53_371x371.png)](https://www.byhand.ai/)\n# [AI by Hand ✍️](https://www.byhand.ai/)\nSubscribeSign in\n# PPO → DPO → GRPO → Rubrics (Mar 2, 2026)\n### AI by Hand ✍️ Seminars\n[![Prof. Tom Yeh's avatar](https://substackcdn.com/image/fetch/$s_!qiqn!,w_36,h_36,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc1c7fa3c-71fa-44a4-bf32-d68e25f2e534_800x800.jpeg)](https://substack.com/@tomyeh)\n[Prof. Tom Yeh](https://substack.com/@tomyeh)\nMar 02, 2026\n∙ Paid\n57\n1\n3\nShare\n###  [Library](https://www.byhand.ai/p/library) › [Seminar Series 2026](https://www.byhand.ai/p/library-series-videos-seminars-2026)\n  1. [Manifold-Constrained Hyper Connections (mHC) from DeepSeek (Jan 9, 2026)](https://www.byhand.ai/p/manifold-constrained-hyper-connections)\n  2. [How Small Models Learn Tool Use (Jan 11, 2026)](https://www.byhand.ai/p/how-small-models-learn-tool-use)\n  3. [Generative AI (Jan 11, 2026)](https://www.byhand.ai/p/introduction-to-gen-ai)\n  4. [Attention (Jan 15, 2026)](https://www.byhand.ai/p/seminar-foundation-attention-to-frontier)\n  5. [Google Ironwood TPU: From Bits to HBM (Jan 19, 2026)](https://www.byhand.ai/p/google-ironwood-tpu-from-bits-to)\n  6. [9 AI Eval Formulas You Must Know (Jan 24, 2026)](https://www.byhand.ai/p/ai-eval-equations-you-must-know)\n  7. [Meta Superintelligence Labs vs Facebook AI Research (Jan 30, 2026)](https://www.byhand.ai/p/meta-superintelligence-labs-vs-facebook)\n  8. [Transformer: Six Levels of Understanding (Feb 13, 2026)](https://www.byhand.ai/p/transformer-six-levels-of-understanding)\n  9. [OpenClaw Seminar (Feb 23, 2026)](https://www.byhand.ai/p/openclaw-seminar)\n  10. **PPO → DPO → GRPO → Rubrics (Mar 2, 2026)**\n  11. [Gemma 4 (May 7, 2026)](https://www.byhand.ai/p/gemma-4-seminar-recording)\n  12. [Qwen 3.6 (May 21, 2026)](https://www.byhand.ai/p/qwen-3-6-seminar-recording)\n  13. [Kimi 3 (Aug 6, 2026)](https://www.byhand.ai/p/library-videos-seminars-2026-kimi-3-seminar-recording)\n\n\nIn last week’s AI by Hand ✍️ seminar, I talked about reinforcement learning from first principles. I started all the way back from _pre-training_ and _inference_ , before climbing forward through the major evolution stages of the modern RL stack: _PPO_ → _DPO_ → _GRPO._ I finally arrived at _Rubrics_ , widely considered one of today’s frontier topics in RL research.\n[![](https://substackcdn.com/image/fetch/$s_!-L9e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20985ed-6155-430c-9f5d-89b7dabaac98_1997x1163.png)](https://substackcdn.com/image/fetch/$s_!-L9e!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20985ed-6155-430c-9f5d-89b7dabaac98_1997x1163.png)\nI was joined by [Cameron R. Wolfe](http://linkedin.com/in/cwolferesearch), author of the _[Deep (Learning) Focus](https://cameronrwolfe.substack.com/)_ newsletter (60K subscribers) and a Senior Research Scientist at Netflix. He gave a special guest lecture based on his popular recent article [Rubric-based Rewards for RL](https://cameronrwolfe.substack.com/p/rubric-rl).\n[![](https://substackcdn.com/image/fetch/$s_!nCxM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5023e359-baf7-4df8-bbb0-9fc2771d9e56_2415x1365.png)](https://substackcdn.com/image/fetch/$s_!nCxM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5023e359-baf7-4df8-bbb0-9fc2771d9e56_2415x1365.png)\nAfter eight seminars in 2026, I’ve settled into a format that has clearly resonated:\nAI by Hand Lecture → Industry Guest Interview → Industry Guest Lecture.\nI’m the professor in the first half—and the student in the second. 🙌\n## Feedback\n[![](https://substackcdn.com/image/fetch/$s_!mL0y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F29f23fa7-5319-4f78-a9bf-e833710297da_1294x1714.png)](https://substackcdn.com/image/fetch/$s_!mL0y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F29f23fa7-5319-4f78-a9bf-e833710297da_1294x1714.png)\n## Industry Expert: Cameron Wolfe\nAs a professor in academia, how can I possibly know what kind of research is really going on in the industry? I turn to people like Cameron. I've been reading his long-form articles long before I even started sharing my own content through this AI by Hand newsletter. Cameron was one of the early supporters who encouraged me to share more publicly. \nI still remember two years ago when he told me he had just gotten married and was moving to Netflix to become a Senior Research Scientist working on reinforcement learning. He mentioned he might need to pause the _Deep Focus_ newsletter for a while. Fortunately for all of us, the break was short. Cameron kept writing. I kept learning.\nIn the seminar, I finally got a chance to interview him so his story can be heard by all of you:\n  * Q: Can you use Jiu-Jitsu as a metaphor to explain reinforcement learning?\n  * Q: Reinforcement learning used to feel like robotics demos. Why has RL become so important for frontier language models?\n  * Q: What are the main challenges of doing RL research at large scale?\n  * Q: What motivated you to write the Deep Focus newsletter?\n\n\nWatch the recording to hear Cameron explain it in his own words.\n## Recording & Workbook\n**Members:** the full recording and the Excel workbook are below ↓\n## This post is for paid subscribers\n[Subscribe](https://www.byhand.ai/subscribe?simple=true&next=https%3A%2F%2Fwww.byhand.ai%2Fp%2Frecording-ppo-dpo-grpo-rubrics&utm_source=paywall&utm_medium=web&utm_content=189703980)\n[Already a paid subscriber? **Sign in**](https://substack.com/sign-in?redirect=%2Fp%2Frecording-ppo-dpo-grpo-rubrics&for_pub=aibyhand&change_user=false)\n© 2026 Tom Yeh · [Privacy](https://substack.com/privacy) ∙ [Terms](https://substack.com/tos) ∙ [Collection notice](https://substack.com/ccpa#personal-data-collected)\n[ Start your Substack](https://substack.com/signup?utm_source=substack&utm_medium=web&utm_content=footer)[Get the app](https://substack.com/app/app-store-redirect?utm_campaign=app-marketing&utm_content=web-footer-button)\n[Substack](https://substack.com) is the home for great culture\n",
  "metadata": {
    "title": "PPO → DPO → GRPO → Rubrics (Mar 2, 2026) - by Prof. Tom Yeh",
    "description": "AI by Hand ✍️ Seminars",
    "keywords": null,
    "author": "Prof. Tom Yeh",
    "og:type": "article",
    "og:title": "PPO → DPO → GRPO → Rubrics (Mar 2, 2026)",
    "og:description": "AI by Hand ✍️ Seminars",
    "og:image": "https://substackcdn.com/image/fetch/$s_!-L9e!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20985ed-6155-430c-9f5d-89b7dabaac98_1997x1163.png",
    "og:url": "https://www.byhand.ai/p/recording-ppo-dpo-grpo-rubrics",
    "twitter:title": "PPO → DPO → GRPO → Rubrics (Mar 2, 2026)",
    "twitter:description": "AI by Hand ✍️ Seminars",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!_VY0!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Faibyhand.substack.com%2Fapi%2Fv1%2Fpost_preview%2F189703980%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2026-08-20T14:29:36.416Z"
  },
  "paywall": {
    "listed_paid": true,
    "detected": true,
    "marker": "this post is for paid subscribers",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "src": "https://substackcdn.com/image/fetch/$s_!fOmN!,w_40,h_40,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F618d59fa-9b8f-4814-8d16-f29c2f23ae53_371x371.png",
      "alt": "AI by Hand ✍️",
      "local_path": "attachments/46f4fdd906da/images/000.png",
      "bytes": 1392,
      "downloaded": true
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!fOmN!",
      "alt": "AI by Hand ✍️",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "w_40",
      "alt": "AI by Hand ✍️",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "h_40",
      "alt": "AI by Hand ✍️",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "c_fill",
      "alt": "AI by Hand ✍️",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "f_auto",
      "alt": "AI by Hand ✍️",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "q_auto:good",
      "alt": "AI by Hand ✍️",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F618d59fa-9b8f-4814-8d16-f29c2f23ae53_371x371.png",
      "alt": "AI by Hand ✍️",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "w_80",
      "alt": "AI by Hand ✍️",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "h_80",
      "alt": "AI by Hand ✍️",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "w_120",
      "alt": "AI by Hand ✍️",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "h_120",
      "alt": "AI by Hand ✍️",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "f_webp",
      "alt": "AI by Hand ✍️",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!qiqn!,w_36,h_36,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc1c7fa3c-71fa-44a4-bf32-d68e25f2e534_800x800.jpeg",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "attachments/46f4fdd906da/images/013.jpeg",
      "bytes": 1210,
      "downloaded": true
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!qiqn!",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "w_36",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "h_36",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "c_fill",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "f_auto",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "q_auto:good",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc1c7fa3c-71fa-44a4-bf32-d68e25f2e534_800x800.jpeg",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "w_72",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "h_72",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "w_108",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "h_108",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "f_webp",
      "alt": "Prof. Tom Yeh's avatar",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!-L9e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20985ed-6155-430c-9f5d-89b7dabaac98_1997x1163.png",
      "alt": "",
      "local_path": "attachments/46f4fdd906da/images/026.png",
      "bytes": 93474,
      "downloaded": true
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!-L9e!",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "w_424",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "c_limit",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "f_auto",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "q_auto:good",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20985ed-6155-430c-9f5d-89b7dabaac98_1997x1163.png",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "w_848",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "w_1272",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "w_1456",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "f_webp",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!nCxM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5023e359-baf7-4df8-bbb0-9fc2771d9e56_2415x1365.png",
      "alt": "",
      "local_path": "attachments/46f4fdd906da/images/037.png",
      "bytes": 78401,
      "downloaded": true
    },
    {
      "src": "https://substackcdn.com/image/fetch/$s_!nCxM!",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "w_424",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "type": "internal",
      "href": "https://www.byhand.ai/"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/library"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/library-series-videos-seminars-2026"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/manifold-constrained-hyper-connections"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/how-small-models-learn-tool-use"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/introduction-to-gen-ai"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/seminar-foundation-attention-to-frontier"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/google-ironwood-tpu-from-bits-to"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/ai-eval-equations-you-must-know"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/meta-superintelligence-labs-vs-facebook"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/transformer-six-levels-of-understanding"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/openclaw-seminar"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/gemma-4-seminar-recording"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/qwen-3-6-seminar-recording"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/p/library-videos-seminars-2026-kimi-3-seminar-recording"
    },
    {
      "type": "internal",
      "href": "https://www.byhand.ai/subscribe?next=https%3A%2F%2Fwww.byhand.ai%2Fp%2Frecording-ppo-dpo-grpo-rubrics&simple=true"
    },
    {
      "type": "external",
      "href": "https://substack.com/@tomyeh"
    },
    {
      "type": "external",
      "href": "https://substackcdn.com/image/fetch/$s_!-L9e!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20985ed-6155-430c-9f5d-89b7dabaac98_1997x1163.png"
    },
    {
      "type": "external",
      "href": "http://linkedin.com/in/cwolferesearch"
    },
    {
      "type": "external",
      "href": "https://cameronrwolfe.substack.com/"
    },
    {
      "type": "external",
      "href": "https://cameronrwolfe.substack.com/p/rubric-rl"
    },
    {
      "type": "external",
      "href": "https://substackcdn.com/image/fetch/$s_!nCxM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5023e359-baf7-4df8-bbb0-9fc2771d9e56_2415x1365.png"
    },
    {
      "type": "external",
      "href": "https://substackcdn.com/image/fetch/$s_!mL0y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F29f23fa7-5319-4f78-a9bf-e833710297da_1294x1714.png"
    },
    {
      "type": "external",
      "href": "https://substack.com/sign-in?change_user=false&for_pub=aibyhand&redirect=%2Fp%2Frecording-ppo-dpo-grpo-rubrics"
    },
    {
      "type": "external",
      "href": "https://substack.com/privacy"
    },
    {
      "type": "external",
      "href": "https://substack.com/tos"
    },
    {
      "type": "external",
      "href": "https://substack.com/ccpa"
    },
    {
      "type": "external",
      "href": "https://substack.com/signup"
    },
    {
      "type": "external",
      "href": "https://substack.com/app/app-store-redirect"
    },
    {
      "type": "external",
      "href": "https://substack.com"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T11:11:38Z",
  "source_url": "https://www.byhand.ai/p/recording-ppo-dpo-grpo-rubrics",
  "delivery_status": "excluded_paid",
  "discovery_status": "excluded_paid",
  "missing_reason": "按原约定排除付费/会员全文"
}