{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Nathan Lambert / Interconnects",
  "title_listed": "5 Useful Things You'll Learn in My New Post-Training Textbook",
  "keywords": "RLHF,PPO,DPO,对齐,人类反馈",
  "domain": "机器学习",
  "content_type": "教程/书籍",
  "is_paid_flag": "否",
  "url": "https://www.interconnects.ai/p/5-useful-things-youll-learn-in-my",
  "publish_date": "2024-04",
  "corpus_stage": "RLHF",
  "notes": "RLHF完整方法论，含PPO/DPO实现细节",
  "id": "1b72e96967cc",
  "collected_at": "2026-10-01T19:50:40",
  "title": "5 useful things you'll learn in my post-training textbook",
  "body_markdown": "Housekeeping: No voiceover on another quick “launch” post. More essays soon!\nAfter a few long years of finding time to document my lessons from training open models, my post-training book is done! It’s published by Manning, under the title\nReinforcement Learning from Human Feedback: Aligning and Post-training LLMs\n.\nTelling the story of the book is a useful way to explain why you may want a copy.\nThe book started as a website where I wanted to document key methods of post-training that had potentially no online material explaining them. If there was something, I couldn’t find it. This existed for more topics than you would expect, given post-training was already popular in 2024 (when I bought the domain), and continues to this day. Topics like\nrejection sampling\n,\noutcome reward models\n, and\ncharacter training\nare prime examples. This has helped make the\nwebsite\nfairly popular, as it’s still one of the few places discussing these topics at a foundational, intuitive way.\nOtherwise, most of the book is about communicating intuitions and history. Much of the LLM industry is defined by core techniques that haven’t changed much in the last few years. This book was my attempt to explain in simple terms why post-training works, what trade-offs people need to make to get it right, and what misconceptions people often get stuck on. To do this, some of the older, foundational blog posts on Interconnects were reworked to stitch together the story behind key mathematical topics. For this reason, a lot of the explanatory text is likely higher voice than your average textbook.\nThis is the book I wanted to read when I was getting started a few years ago! With how many people still ask me basic post-training questions, in fact a population that’s accelerating in size, I suspect this book will be very well received. I still use the book regularly and hear from established researchers all over the industry that they do too. So, it’s not a beginner book — it’s more tailored to someone who has already finished a bachelor’s degree in CS — but if you master it you will be far ahead in your post-training worldview.\nShare\nA discount to readers!\nThe book is also freely available\nonline\nand comes with a full 12 hour course (\nslides\n+\nvideo on YouTube\n), a simple\ncode-base\nwith suggested exercises, and model completion comparisons. It’s 50% off until August 19th on Manning with the code\nPBLambert\n.\nBuy my book from Manning\nBuy my book on Amazon\nYes, the title of this book is a little outdated — I’ve learned some lessons and fought some battles with the publishing process — but I can guarantee the content is very fresh. I regularly reference the book for my research work, and hear from friends that do the same. I added a section on on-policy distillation at the last possible moment! The book is shipping from Manning and Amazon US now, and from Amazon UK in October.\n1. Intuitions for how RL algorithms change the outputs of models\nBy word or page count, the book is about 25% RL. This seems appropriate. If there’s one thing the book is doing it’s teaching people how to think about various RL algorithms. This intuition, from the policy-gradient theorem to PPO to modern versions like GSPO and CISPO, are crucial to understanding if a new algorithm is fake or has potential (no new algorithm will be proven right out of the gates).\nBelow is an example intuition you should be able to follow\nafter\nreading.\nFor example, here’s a fun figure that we’ve iterated on for the PPO clipping understanding. At the end of the day, PPO’s surrogate objective reduces to six regions. These can be seen as two gradients, when the advantage for a token is positive or negative, depending on the current value of the policy ratio. For an individual sample in a completion, it lives somewhere on this plot. If it was the first gradient step in the batch, it starts at 1 on the x axis (gradient always flows), then depending how the ratio updates after changing the RL policy behavior, the gradient is either the same or becomes 0 (which is what the clipping arguments are for).\nThis intuition filters very closely into how the systems are designed, in order to manage the gradients and numerical issues they tend to cause. The math-focused policy-gradient section is pretty thorough, covering all the algorithms you’ve likely heard about in the last 3 years:\nDeriving the Policy Gradient\nVanilla Policy Gradient\nREINFORCE\nREINFORCE Leave One Out (RLOO)\nProximal Policy Optimization (PPO)\nUnderstanding the PPO Objective\nValue Functions and PPO\nGroup Relative Policy Optimization (GRPO)\nGroup Sequence Policy Optimization (GSPO)\nClipped Importance Sampling Policy Optimization (CISPO)\nComparing Algorithms\n2. An understanding of the crucial factors facing new RL systems and algorithms\nMost of modern RL is a systems problem balancing a few problems — how off-policy the data is, training-inference mismatch, and throughput. The core systems design, asynchronous RL with separate GPUs for the learners (the GPUs which take gradient steps) and actors (the GPUs which generate the rollouts in the environment), has been similar for a few years.\nAgentic tasks are only adding more infrastructure on top of these fundamentals. The book is designed to be the simplest resource to start from roughly 0 LLM RL knowledge and be ready to tinker with the systems. It starts with basics, such as explaining the general form of implementing an RL algorithm:\npg_loss = -advantages * ratio\nIt continues with teaching you about loss aggregation — the idea that spawned DAPO and Dr. GRPO as some of the seminal, early GRPO variants — and truncated importance sampling — the technique used to make PPO work in early RL experiments.\nPolicy-Gradient Basics\nLoss Aggregation Tradeoffs\nAsynchronous RL Systems\nTruncated Importance Sampling\nExample: PPO\nExample: GRPO\nShare\n3. The histories that lead to modern post-training\nKnowing how a field came to be has always been fascinating to me. As you become an expert, knowing the history of your field better than anyone is what lets you make the best predictions (Bill Gurley gives similar advice in his recent\nbook\n). In a time when the foundations of deep learning were being built, like the transformer, the core of modern post-training was also born in the alignment field. The book will walk you through 3 eras, when\nresearchers learned to do RL on preferences\ngenerally until ~2018, spent a few years learning\nhow to apply it to language models\nfrom 2019 to 2022, and from 2023 on exploited the examples set by ChatGPT. Much like those early to scaling LLMs, the people who created this field a decade ago deserve incredible credit for how modern progress has unfolded.\nChapter 2 of the book is a crash course on this, but the book is littered with this type of thinking.\n4. Dispelling the magic of “distillation”\nIt’s really nice to have a boring textbook\nchapter\non distillation given the broader AI policy discussions ongoing. This one, chapter 12, explains the various industry-standard ways that outputs from an LLM are used to train downstream models. When the technique is often described in nefarious ways, and as a tool of geopolitical competition, it’s a deescalatory action to break out a 300 page textbook to explain to someone how broad the term they’re attacking is.\nWith this, chapters 10 through 12 are all about making some opaque practices of the data industries clearer to readers.\nThe distillation chapter continues the theme I outlined above, explaining the key changes that needed to be made to transition the early knowledge distillation literature of 2015 to 2-3 key insights that got us to the multi-teacher on-policy distillation (MOPD) of models like Xiaomi MiMo-V2-Flash and DeepSeek V4.\n5. A survey of all the other little headaches you encounter when trying to do post-training right\nThe second half of this book goes into a tour of over-optimization, regularization, evaluation, and character training, which is all about the many ways post-training can go wrong and what you need to stay on top of. This is what differentiates the book most from those that are just a list of code exercises and equations, but it explains things like\nwhy RL generalizes when SFT forgets\n(at the math level) or which techniques frontier labs use to\nshape the personalities of the models\nand\nwhy those often go too far\n. The book presents the tools you will use and then opens up the floodgates of all the challenges you’re going to face when you actually try to put them to use.\nAs I wrote the takeaways for the book, I was reminded of an old piece of advice I gave for the AI era of building companies, and how\npeople\ndo\nneed to care about research right now\n. With the pace of progress in AI, the time it takes for a research paper to land in a frontier model is 3-9months. Previously in big tech, that would be years, so it was fine to take a hands off approach to new research. For people with companies relying on being at the frontier in specific niches of LLMs, the dynamic today can make or break the company. This book is useful because it trains you at understanding which research matters — it helps you develop research taste.\nIf none of this resonates with you, you should buy my book because it makes me happy and I work very hard on all of this.\nI really like this photo.",
  "metadata": {
    "title": "5 useful things you'll learn in my post-training textbook",
    "description": "After a few long years of finding time to document my lessons from training open models, my post-training book is done!",
    "keywords": null,
    "author": "Nathan Lambert",
    "og:type": "article",
    "og:title": "5 useful things you'll learn in my post-training textbook",
    "og:description": "After a few long years of finding time to document my lessons from training open models, my post-training book is done!",
    "og:image": "https://substackcdn.com/image/fetch/$s_!ccYc!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4468909c-7481-4b27-9566-6fc838b95743_1074x731.jpeg",
    "og:url": "https://www.interconnects.ai/p/5-useful-things-youll-learn-in-my",
    "twitter:title": "5 useful things you'll learn in my post-training textbook",
    "twitter:description": "After a few long years of finding time to document my lessons from training open models, my post-training book is done!",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!1GME!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Frobotic.substack.com%2Fapi%2Fv1%2Fpost_preview%2F208127655%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2026-08-10T13:05:51.205Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!L6n5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6018ee47-c200-4a7e-acbf-8b79dbd86495_4155x3893.png",
      "bytes": 319900,
      "reused": true,
      "local_path": "attachments/1b72e96967cc/images/027.png",
      "src": "https://substackcdn.com/image/fetch/$s_!L6n5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6018ee47-c200-4a7e-acbf-8b79dbd86495_4155x3893.png",
      "alt": "Figure 5: Visualization of the PPO objective J(\\theta) as a function of the policy ratio \\rho(\\theta), for both positive and negative advantage. Within each panel, the three ratio regions are annotated with their unclipped term, clipped term, resulting objective, and gradient.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!IlYo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff930eabc-71b0-4f2f-9126-d88673466c80_3835x2119.png",
      "bytes": 172605,
      "reused": true,
      "local_path": "attachments/1b72e96967cc/images/038.png",
      "src": "https://substackcdn.com/image/fetch/$s_!IlYo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff930eabc-71b0-4f2f-9126-d88673466c80_3835x2119.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!wzyr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa96b87d7-181d-4dc0-9a5b-6b1e5fc31675_2401x552.png",
      "bytes": 52951,
      "reused": false,
      "local_path": "attachments/1b72e96967cc/images/body-53c3e089ac5933d8c475.png",
      "src": "https://substackcdn.com/image/fetch/$s_!wzyr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa96b87d7-181d-4dc0-9a5b-6b1e5fc31675_2401x552.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!ccYc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4468909c-7481-4b27-9566-6fc838b95743_1074x731.jpeg",
      "bytes": 91274,
      "reused": false,
      "local_path": "attachments/1b72e96967cc/images/body-73307b641a71c012d662.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!ccYc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4468909c-7481-4b27-9566-6fc838b95743_1074x731.jpeg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=jQPiH-KB4B0",
      "status": "ok",
      "local_path": "attachments/1b72e96967cc/videos/jQPiH-KB4B0.mp4",
      "bytes": 8191216,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "215.141000"
    }
  ],
  "links": [
    {
      "href": "https://rlhfbook.com/c/09-rejection-sampling",
      "label": "rejection sampling",
      "type": "a"
    },
    {
      "href": "https://rlhfbook.com/c/05-reward-models",
      "label": "outcome reward models",
      "type": "a"
    },
    {
      "href": "https://rlhfbook.com/c/17-product",
      "label": "character training",
      "type": "a"
    },
    {
      "href": "https://rlhfbook.com/",
      "label": "website",
      "type": "a"
    },
    {
      "href": "https://www.interconnects.ai/p/5-useful-things-youll-learn-in-my?utm_source=substack&utm_medium=email&utm_content=share&action=share",
      "label": "Share",
      "type": "a"
    },
    {
      "href": "https://rlhfbook.com/course",
      "label": "slides",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=jQPiH-KB4B0&list=PLL1tdVxB1CpVpEtMHxwuR4uI4Lxjw00_y&pp=sAgC",
      "label": "video on YouTube",
      "type": "a"
    },
    {
      "href": "https://github.com/natolambert/rlhf-book/tree/main/code",
      "label": "code-base",
      "type": "a"
    },
    {
      "href": "https://hubs.la/Q03TsMBq0",
      "label": "Buy my book from Manning",
      "type": "a"
    },
    {
      "href": "https://amzn.to/4cwCDJQ",
      "label": "Buy my book on Amazon",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!L6n5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6018ee47-c200-4a7e-acbf-8b79dbd86495_4155x3893.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://rlhfbook.com/c/06-policy-gradients",
      "label": "Deriving the Policy Gradient",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!IlYo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff930eabc-71b0-4f2f-9126-d88673466c80_3835x2119.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Runnin-Down-Dream-Thrive-Actually/dp/0593799666",
      "label": "book",
      "type": "a"
    },
    {
      "href": "https://rlhfbook.com/c/02-related-works",
      "label": "researchers learned to do RL on preferences",
      "type": "a"
    },
    {
      "href": "https://rlhfbook.com/c/12-synthetic-data",
      "label": "chapter",
      "type": "a"
    },
    {
      "href": "https://rlhfbook.com/c/15-regularization",
      "label": "why RL generalizes when SFT forgets",
      "type": "a"
    },
    {
      "href": "https://rlhfbook.com/c/14-over-optimization",
      "label": "why those often go too far",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!ccYc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4468909c-7481-4b27-9566-6fc838b95743_1074x731.jpeg",
      "label": "",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:02Z",
  "source_url": "https://www.interconnects.ai/p/5-useful-things-youll-learn-in-my",
  "papers": [],
  "actual_video_files": [
    {
      "local_path": "attachments/1b72e96967cc/videos/jQPiH-KB4B0.mp4",
      "bytes": 8191216,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "215.141000",
      "status": "ok"
    }
  ],
  "body_chars": 9340,
  "expected_body_images": 4,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/1b72e96967cc/pages/source.html",
  "local_html": "attachments/1b72e96967cc/pages/article.html",
  "body_base_path": "attachments/1b72e96967cc/pages/",
  "body_markdown_path": "attachments/1b72e96967cc/pages/article.md"
}