{ "source_file": "0930", "source_platform": "Substack", "author": "Nathan Lambert / Interconnects", "title_listed": "Why Reasoning Models Will Generalize", "keywords": "推理模型,泛化,RLHF,过程奖励", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://www.interconnects.ai/p/why-reasoning-models-will-generalize", "publish_date": "2025-01", "corpus_stage": "RLHF", "notes": "论证推理模型泛化能力，含RL训练方法分析", "id": "17154eec3651", "collected_at": "2026-10-01T19:50:22", "title": "Why reasoning models will generalize - by Nathan Lambert", "body_markdown": "This post is early to accommodate some last minute travel on my end!\nThe new models trained to express extended chain of thought are going to generalize outside of their breakthrough domains of code and math. The “reasoning” process of language models that we use today is chain of thought reasoning. We ask the model to work step by step because it helps it manage complexity, especially in domains where the answer requires precision across multiple specific tokens. The domains where\nchain of thought\n(CoT) is most useful today are code, mathematics, and other “reasoning” tasks\n1\n. These are the domains where models like o1, R1, Gemini-Thinking, etc. were designed for.\nDifferent intelligences reason in different ways that correspond to how they store and manipulate information. Humans compress a lifetime of experience into our spectacular, low-power brains that draw on past experience almost magically. The words that follow in this blog are also autoregressive, like the output of a language model, but draw on hours and hours of background processing as I converge on this argument.\nLanguage models, on the other hand, are extremely general and do not today have architectures (or use-cases) that continually re-expose them to relevant problems and fold information back in a compressed form. Language models are very large, sophisticated, parametric probability distributions. All of their knowledge and information processing\npower\nis stored in the raw weights. Therein, they need a way of processing information that matches this. Chain of thought\nis that alignment\n.\nChain of thought reasoning allows information to be naturally processed in smaller chunks, allowing the large, brute force probability distribution to work one token at a time. Chain of thought, while allowing more compute per important token, also allows the models to\nstore\nintermediate information in their context window without needing explicit recurrence.\nRecurrence is required for reasoning and this can either happen in the parameter or state-space. Chain of thoughts with transformers handles all of this in the state-space of the problems. The humans we look at as the most intelligent have embedded information directly in the parameters of our brains that we can draw on.\nHere is the only assumption of this piece —\nchain of thought is a natural fit for language models to “reason” and therefore one should be optimistic about training methods that are designed to enhance it generalizing to many domains.\n2\nBy the end of 2025 we should have ample evidence of this given the pace of the technological development.\nShare\nIf the analogies of\ntypes\nof intelligence aren’t convincing enough, a far more practical way to view the new style of training is a method that teaches the model to be better at allocating more compute to harder problems. If the skill is compute allocation, it is fundamental to the models handling a variety of tasks. Today’s reasoning models do not solve this perfectly, but they open the door for doing so precisely.\nThe nature of this coming\ngeneralization\nis not that these models are one size fits all, best in all cases: speed, intelligence, price, etc. There’s still no free lunch. A\nrealistic\noutcome for reasoning heavy models in the next 0-3 years is a world where:\nReasoning trained models are superhuman on tasks with verifiable domains, like those with initial progress: Code, math, etc.\nReasoning trained models are well better in\npeak\nperformance than existing autoregressive models in many domains we would not expect and are not necessarily verifiable.\nReasoning trained models are still\nbetter\nin performance at the long-tail of tasks, but\nworse\nin cost given the high inference costs of long-context.\nMany of the leading figures in AI have been saying for quite some time that powerful AI is going to be “spikey\" when it shows up — meaning that the capabilities and improvements will vary substantially across domains — but encountering this reality is very unintuitive.\nSome evidence for generalization of reasoning models already exists.\nOpenAI has already published multiple safety-oriented research projects with their new reasoning models in\nDeliberative Alignment: Reasoning Enables Safer Language Models\nand\nTrading Inference-Time Compute for Adversarial Robustness\n. These papers show their new methods can be translated to various safety domains, i.e. model safety policies and jailbreaking. The deliberative alignment paper shows them integrating a softer reward signal into the reasoning training — having a language model check how the safety policies apply to outputs.\nAn unsurprising quote from the deliberative alignment release related to generalization:\nwe find that deliberative alignment enables strong generalization to out-of-distribution safety scenarios.\nSafety, qualitatively, is very orthogonal to traditional reasoning problems. Safety is very subjective to the information provided and subtle context, where math and coding problems are often about many small, forward processing steps towards a final goal. More behaviors will fit in between those.\nThis generative verifier for safety is not a ground truth signal and\ncould\ntheoretically be subject to reward hacking, but it was avoided. Generative verifiers will be crucial to expanding this training to countless domains — they’re easy to use and largely a new development. The field of LLM-as-a-judge (and related synthetic data pipelines) only really became stable with models at the level of GPT-4.\nReasoning models trained as a judge are a very natural fit because the exact token for a predicted reward or ranking is crucial — CoT is essential. All of the progress here relies on continued progress on both generators and verifiers. o1 et al. were likely trained with mostly explicit, code verifiers. They spawned far more powerful generators, which will enable new types of verifiers. Then, we can train better models (and so on).\nOnto another example of unexpected performance of new reasoning trained models.\nDeepSeek-R1\n, the new open-weight o1 replication has been showing up at the top of many random benchmarks as top overall, above Claude 3.5 Sonnet, Gemini, and GPT-4o, and alongside o1. Examples include a\ncreative writing and humor leaderboard\nor the brand-new, extremely challenging benchmark from the Center for AI Safety and Scale AI —\nHumanity’s Last Exam\n. Oh, and yes, it’s best on both accuracy and the new metric “calibration error” which is designed to have the model express its own uncertainty. Calibration is a long-sought behavior in traditional LMs and turns out maybe reasoning training helps it?\nA lot of my friends find o1-pro to be clearly the most useful AI model in their daily workflows (one example\nhere\nand a similar R1 example\nhere\n).\nChatBotArena\nhas all of the new models, from o1, Gemini-Thinking, and R1 as some of the top models these organizations have in the best “normal use” evaluation the AI community has.\n3\nThese reasoning models are definitely absorbing the other lessons learned in post-training across the AI industry.\nThe explosion of R1 caused arguably the biggest general awareness of AI moment since the original ChatGPT. DeepSeek’s App has been the number one overall free app in the U.S. and non-technical users are getting meaningful value out of\nseeing the reasoning process\n. What was a niche training process is bringing many more types of benefits than expected.\nAll of this is just on “day 1” of this technology. Reasoning models are going to proceed at a rate far, far faster than most expect.\nThese models will not be state-of-the-art on every domain, but probably far more than you expect. Language models are a complex technology and they will never be one size fits all, but the ground is being reshaped under us.\nEspecially, where the standard models match the reasoning models abilities, you’ll be paying way more for the same performance. At the same time, so many domains are going to be open to the “if you pay a little bit more, the reasoning model will get you a bit more performance,” which will accrue so much value over time.\nThese are trade-offs that many in the AI industry see at face value. Many ask where Anthropic’s reasoning model is, but they may never explicitly have one. Before o1 launched, Claude was\nalready using extra tokens hidden from the user to improve the quality of responses\n. Anthropic CEO Dario Amodei commented on their approach in an\ninterview with Joanna Stern\nof the WSJ recently:\nTo say a little about reasoning models, our perspective is a little different, which is that there’s been this whole idea of reasoning models and test-time compute as if they’re a totally different way of doing things. That’s not our perspective. We see it more as a continuous spectrum — the ability for models to think, reflect on their own thinking, and ultimately produce a result. If you use Sonnet 3.5, sometimes it already does that to some extent. But I think the change we’re going to see is a larger-scale use of reinforcement learning, and when you train the model with reinforcement learning, it starts to think and reflect more.\nIt’s not like reasoning or test-time compute — or whatever it’s called — is a totally new method. It’s more like an emergent property, a consequence of training the model in an outcome-based way at a larger scale. I think\nthat will lead to something that continuously interpolates between reasoning and other tasks, fluidly combining reasoning with everything else models do.\nAs you’ve said, we’ve often focused on making sure using the model is a smooth experience, allowing people to get the most out of it. I think with reasoning models, we may take a similar approach and do something different from what others are doing.\nThe newest Claude 3.5 Sonnet models are very likely already trained\nto some extent\nwith RL on verifiable outcomes. Just days before o1 was launched,\nClaude’s behavior of “I’m thinking about that”\nwas the biggest indicator we had of consumer companies trading more compute for better responses. Anthropic hasn’t shifted their strategy here, and you can decide how much weight you want to put in their CEO’s recent comments.\nInterconnects is a reader-supported publication. Consider becoming a subscriber.\nSubscribe\nThe techniques are here to stay, and it took revolutionary new models to show us that.\nLike many new technologies, we needed to be shown what was possible, and then it can be\nfolded back into normal experience\n. o1 was this breakthrough, and the benefits of reasoning training will now expand out into all of the AI products we are using day and night.\nTo end, I leave you with a quote from the\nDeepSeek R1\npaper, where the authors reflect on their experience with the model(s):\nOne of the most remarkable aspects of this self-evolution is the emergence of sophisticated behaviors as the test-time computation increases. Behaviors such as reflection—where the model revisits and reevaluates its previous steps—and the exploration of alternative approaches to problem-solving arise spontaneously. These behaviors are not explicitly programmed but instead emerge as a result of the model’s interaction with the reinforcement learning environment. This spontaneous development significantly enhances DeepSeek-R1-Zero’s reasoning capabilities, enabling it to tackle more challenging tasks with greater efficiency and accuracy.\nThanks to Ross Taylor and Hamish Ivison, for discussions that helped inspire this post.\n1\na good example of a “reasoning” benchmark that is different is BigBenchHard that tests things like multi-step reasoning, logic, abstract algebra, etc.\n2\nBy domains I’m focused on text domains. I expect improvements across all modalities, but how it is translated is less clear to me.\n3\nWhich will be\ndriving a price war\n.", "metadata": { "title": "Why reasoning models will generalize - by Nathan Lambert", "description": "People underestimate the long-term potential of “reasoning.”", "keywords": null, "author": "Nathan Lambert", "og:type": "article", "og:title": "Why reasoning models will generalize", "og:description": "People underestimate the long-term potential of “reasoning.”", "og:image": "https://substackcdn.com/image/fetch/$s_!9960!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a17cb48-90d9-4e21-8ec8-4c9141384155_1280x720.png", "og:url": "https://www.interconnects.ai/p/why-reasoning-models-will-generalize", "twitter:title": "Why reasoning models will generalize", "twitter:description": "People underestimate the long-term potential of “reasoning.”", "twitter:image": "https://substackcdn.com/image/fetch/$s_!6_A7!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Frobotic.substack.com%2Fapi%2Fv1%2Fpost_preview%2F155503276%2Ftwitter.jpg%3Fversion%3D4", "twitter:card": "summary_large_image", "article:modified_time": "2025-01-28T21:05:14.145Z" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!V-3i!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc725b9b4-1af0-439e-86ad-b6554415961d_1102x830.png", "bytes": 71776, "reused": true, "local_path": "attachments/17154eec3651/images/027.png", "src": "https://substackcdn.com/image/fetch/$s_!V-3i!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc725b9b4-1af0-439e-86ad-b6554415961d_1102x830.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!TDWr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6af88ffd-6927-4af8-adec-e6d1a6bbc908_1206x903.jpeg", "bytes": 124396, "reused": true, "local_path": "attachments/17154eec3651/images/038.jpeg", "src": "https://substackcdn.com/image/fetch/$s_!TDWr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6af88ffd-6927-4af8-adec-e6d1a6bbc908_1206x903.jpeg", "alt": "", "downloaded": true } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/watch?v=snkOMOjiVOk", "status": "ok", "local_path": "attachments/17154eec3651/videos/snkOMOjiVOk.mp4", "bytes": 479917810, "valid": true, "height": 1080, "width": 1920, "duration": "2132.921000" } ], "links": [ { "href": "https://arxiv.org/abs/2201.11903", "label": "chain of thought", "type": "a" }, { "href": "https://www.interconnects.ai/p/why-reasoning-models-will-generalize?utm_source=substack&utm_medium=email&utm_content=share&action=share", "label": "Share", "type": "a" }, { "href": "https://arxiv.org/abs/2412.16339", "label": "Deliberative Alignment: Reasoning Enables Safer Language Models", "type": "a" }, { "href": "https://cdn.openai.com/papers/trading-inference-time-compute-for-adversarial-robustness-20250121_1.pdf", "label": "Trading Inference-Time Compute for Adversarial Robustness", "type": "a" }, { "href": "https://www.interconnects.ai/p/deepseek-r1-recipe-for-o1", "label": "DeepSeek-R1", "type": "a" }, { "href": "https://eqbench.com/creative_writing.html", "label": "creative writing and humor leaderboard", "type": "a" }, { "href": "https://agi.safe.ai/", "label": "Humanity’s Last Exam", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!V-3i!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc725b9b4-1af0-439e-86ad-b6554415961d_1102x830.png", "label": "", "type": "a" }, { "href": "https://x.com/deanwball/status/1871396913473335701", "label": "here", "type": "a" }, { "href": "https://substack.com/@jasmine/note/c-88814858", "label": "here", "type": "a" }, { "href": "https://www.interconnects.ai/p/chatbotarena-the-future-of-llm-evaluation", "label": "ChatBotArena", "type": "a" }, { "href": "https://x.com/splitbycomma/status/1883588991813042605", "label": "seeing the reasoning process", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!TDWr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6af88ffd-6927-4af8-adec-e6d1a6bbc908_1206x903.jpeg", "label": "", "type": "a" }, { "href": "https://www.interconnects.ai/i/148388607/self-talk-in-language-models", "label": "already using extra tokens hidden from the user to improve the quality of responses", "type": "a" }, { "href": "https://www.youtube.com/watch?v=snkOMOjiVOk", "label": "interview with Joanna Stern", "type": "a" }, { "href": "https://x.com/morqon/status/1881770911075873054", "label": "folded back into normal experience", "type": "a" }, { "href": "https://www.latent.space/p/reasoning-price-war", "label": "driving a price war", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:47:01Z", "source_url": "https://www.interconnects.ai/p/why-reasoning-models-will-generalize", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/2201.11903", "bytes": 891773, "reused": false, "local_path": "papers/repair/52cc4c4f6cf368123b97.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2412.16339", "bytes": 2439335, "reused": false, "local_path": "papers/repair/922095b1df96b6bd1de7.pdf" }, { "status": "ok", "url": "https://cdn.openai.com/papers/trading-inference-time-compute-for-adversarial-robustness-20250121_1.pdf", "bytes": 4805418, "reused": false, "local_path": "papers/repair/d9e50457fb270435d481.pdf" } ], "actual_video_files": [ { "local_path": "attachments/17154eec3651/videos/snkOMOjiVOk.mp4", "bytes": 479917810, "valid": true, "height": 1080, "width": 1920, "duration": "2132.921000", "status": "ok" } ], "body_chars": 11824, "expected_body_images": 2, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/17154eec3651/pages/source.html", "local_html": "attachments/17154eec3651/pages/article.html", "body_base_path": "attachments/17154eec3651/pages/", "body_markdown_path": "attachments/17154eec3651/pages/article.md" }