{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Nathan Lambert / Interconnects",
  "title_listed": "Scaling Realities: What LLM Scaling Laws Miss",
  "keywords": "Scaling Laws,计算最优,数据配比",
  "domain": "大模型算法",
  "content_type": "研究综述",
  "is_paid_flag": "否",
  "url": "https://www.interconnects.ai/p/scaling-realities",
  "publish_date": "2025-02",
  "corpus_stage": "预训练",
  "notes": "批判性分析Scaling Laws的局限与实际偏差",
  "id": "ec6aca83c274",
  "collected_at": "2026-10-01T19:50:27",
  "title": "Scaling realities - by Nathan Lambert - Interconnects AI",
  "body_markdown": "Notes: this is a cleaned version of a long-post I shared on socials earlier this week. Second, there are a few final edits for clarity not reflected in the voiceover.\nThere is a lot of discussion on whether \"scaling is done,\" with\nstories from The Information\nsaying that the latest GPT models aren't showing what OpenAI wanted while Sam Altman still parades around saying AGI is near (and a\nsimilar, more detailed story\nfrom Bloomberg on Gemini). We’re still waiting for Gemini 1.5 Ultra and Claude 3.5 Opus, let alone the next indexing.\nBoth narratives can be true:\nScaling is still working at a technical level.\nThe rate of improvement for users is slowing. Especially the average user.\nWithout the performance gains clearly visible to users, launching the next scale of models is a strategic gamble.\nUltimately, a scaling law is a technical definition between test loss and training compute. This test loss does not necessarily need to be correlated with performance — the central motivation of my recent post on scaling's impacts, but one that got caught up in a different focus on the AGI strawman.\nHow scaling changes model behavior\nNathan Lambert\n·\nOctober 9, 2024\nRead full story\nThis post is just about one specific reality:\nScaling works and that doesn’t solve all the economic problems for AI\n.\nSubscribe\nInside the labs, I've heard nothing to say that scaling laws are stopping. If anything, the curves are starting to bend slightly away from a linear-looking power law, but by spending more, we're straightening them out.\nThe biggest blockers to\ntraining\nthese models to test the scaling laws are physical constraints — GPU allocation, power supply, permitting, etc. Some of these are modeling questions, like how to batch together training updates from multiple data centers or how to get more usable synthetic training data, but most of them are not related to the definition of scaling laws.\nEven with scaling laws \"working,” the perception of the final post-trained GPT-5, Claude 4, Gemini 2 class models can be underwhelming. This is because our goalposts are based on certain types of chat tasks.\nA certain psychological conditioning happens for the best language models now, which has happened forever with humans. Because a language model, or human, sounds smart, we assume they can accomplish a diverse spectrum of tasks. Language models are getting robustness in capabilities we’ve largely already attributed to them. We’re backfilling and pushing forward.\nFor general chat tasks, the models went from unusable to compelling but buggy with GPT 3.5, the o.g. ChatGPT. With GPT-4, normal chat tasks and coding and math became viable. With GPT-5, what is it expected to unlock? It's not clear to me what a better chat model would look like. Robustness is hard to get excited about.\nOpenAI has backed itself into a corner with its messaging. Their messaging around AGI soon made people expect this to be in a form factor they know. Really, the AGI they're building is not contingent on GPT-5 being mind-blowing. It's a system that will use GPT-5 as one part of it. It’ll be agents, like computer use, or even things that are entirely new form factors — agents that are dispatched on the web. These views of “AGI” are entirely separate from what ChatGPT is.\nAll of this is to say that I find both arguments to be true. Scaling is working, as the AI true believers  keep saying whatever the media environment is. Still, the product limits of current ChatGPT forms are hitting limits where a bigger model may not be the answer. We need more specialized models.\nOpenAI’s o1\nis one of the first forays into \"something different.\" No one is making noises about that capping out this year or next.\nWith this, I still strongly believe there are way more products to be built with current AI that will unlock immense value. None of these trends doom the industry at large, but investments will slow if OpenAI et al. need to change their fundraising strategy.\nWe still have a huge capability overhang. Our ability to create specific capabilities in models is still growing very fast (e.g. by the expansion of post-training). I see a ton of low-hanging fruit and see no reason to be pessimistic about the trajectory of AI improvement, it is just important to ground where it is and what it is. This capability-product overhang takes two forms:\nPowerful AI models can do a ton of things today that either a) we don’t know or b) we don’t know other people will value.\nScaling will keep making the models “better” in a technical sense, and we have no way to know what this may unlock.\nAll of us should be excited that companies are willing to risk their economic viability to explore this frontier. The benefits they find will last forever, whether or not the first batch of companies does.\nShare\nThanks to\nAlberto Romero\nat\nThe Algorithmic Bridge\nand many\nfriends in the Interconnects paid Discord server for discussions that cultivated this post. The recent\nLex interview with Dario\nhas more useful commentary on the topic. If you like this type of shorter and to the point post, let me know!",
  "metadata": {
    "title": "Scaling realities - by Nathan Lambert - Interconnects AI",
    "description": "Both stories are true. Scaling still works. OpenAI et al. still have oversold their promises.",
    "keywords": null,
    "author": "Nathan Lambert",
    "og:type": "article",
    "og:title": "Scaling realities",
    "og:description": "Both stories are true. Scaling still works. OpenAI et al. still have oversold their promises.",
    "og:image": "https://substackcdn.com/image/fetch/$s_!7bGC!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9992bd68-3457-4e1c-95fa-dc8a06a49ed8_1280x720.png",
    "og:url": "https://www.interconnects.ai/p/scaling-realities",
    "twitter:title": "Scaling realities",
    "twitter:description": "Both stories are true. Scaling still works. OpenAI et al. still have oversold their promises.",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!h7cw!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Frobotic.substack.com%2Fapi%2Fv1%2Fpost_preview%2F151557775%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2025-10-19T23:06:31.960Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!A-hv!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71b55393-74ea-47c9-8331-1e14d3fc5330_2054x1738.png",
      "bytes": 4979,
      "reused": true,
      "local_path": "attachments/ec6aca83c274/images/027.png",
      "src": "https://substackcdn.com/image/fetch/$s_!A-hv!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71b55393-74ea-47c9-8331-1e14d3fc5330_2054x1738.png",
      "alt": "How scaling changes model behavior",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!e6sL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb995bfdc-70df-437e-bdb5-c5cc616bf6ec_1280x720.png",
      "bytes": 13551,
      "reused": true,
      "local_path": "attachments/ec6aca83c274/images/036.png",
      "src": "https://substackcdn.com/image/fetch/$s_!e6sL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb995bfdc-70df-437e-bdb5-c5cc616bf6ec_1280x720.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=ugvHCXCOmm4",
      "status": "ok",
      "local_path": "attachments/ec6aca83c274/videos/ugvHCXCOmm4.mp4",
      "bytes": 1433426886,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "18900.041000"
    }
  ],
  "links": [
    {
      "href": "https://www.theinformation.com/articles/openai-shifts-strategy-as-rate-of-gpt-ai-improvements-slows",
      "label": "stories from The Information",
      "type": "a"
    },
    {
      "href": "https://www.bloomberg.com/news/articles/2024-11-13/openai-google-and-anthropic-are-struggling-to-build-more-advanced-ai",
      "label": "similar, more detailed story",
      "type": "a"
    },
    {
      "href": "https://www.interconnects.ai/p/how-scaling-changes-model-behavior",
      "label": "How scaling changes model behavior Nathan Lambert · October 9, 2024 Read full story",
      "type": "a"
    },
    {
      "href": "https://substack.com/profile/10472909-nathan-lambert",
      "label": "Nathan Lambert",
      "type": "a"
    },
    {
      "href": "https://www.interconnects.ai/p/reverse-engineering-openai-o1",
      "label": "OpenAI’s o1",
      "type": "a"
    },
    {
      "href": "https://www.interconnects.ai/p/scaling-realities?utm_source=substack&utm_medium=email&utm_content=share&action=share",
      "label": "Share",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!e6sL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb995bfdc-70df-437e-bdb5-c5cc616bf6ec_1280x720.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://open.substack.com/users/91075008-alberto-romero?utm_source=mentions",
      "label": "Alberto Romero",
      "type": "a"
    },
    {
      "href": "https://open.substack.com/pub/thealgorithmicbridge",
      "label": "The Algorithmic Bridge",
      "type": "a"
    },
    {
      "href": "https://youtu.be/ugvHCXCOmm4",
      "label": "Lex interview with Dario",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:01Z",
  "source_url": "https://www.interconnects.ai/p/scaling-realities",
  "papers": [],
  "actual_video_files": [
    {
      "local_path": "attachments/ec6aca83c274/videos/ugvHCXCOmm4.mp4",
      "bytes": 1433426886,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "18900.041000",
      "status": "ok"
    }
  ],
  "body_chars": 5100,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ec6aca83c274/pages/source.html",
  "local_html": "attachments/ec6aca83c274/pages/article.html",
  "body_base_path": "attachments/ec6aca83c274/pages/",
  "body_markdown_path": "attachments/ec6aca83c274/pages/article.md"
}