{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Deep (Learning) Focus Cameron R. Wolfe",
  "title_listed": "LLM Scaling Laws Explained",
  "keywords": "Scaling Law,扩展规律",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://cameronrwolfe.substack.com/p/llm-scaling-laws",
  "publish_date": "2024-04",
  "corpus_stage": "预训练/SFT",
  "notes": "LLM扩展规律解析",
  "id": "514a31d31a2b",
  "collected_at": "2026-10-01T20:48:02",
  "title": "Scaling Laws for LLMs: From GPT-3 to o3",
  "body_markdown": "(from [1, 7, 10, 21])\nA majority of recent advancements in AI research—\nand large language models (LLMs) in particular\n—have been driven by scale. If we train larger models over more data, we get better results. This relationship can be defined more rigorously via a scaling law, which is just an equation that describes how an LLM’s test loss will decrease as we increase some quantity of interest (e.g., training compute). Scaling laws help us to predict the results of larger and more expensive training runs, giving us the necessary confidence to continue investing in scale.\n“If you have a large dataset and you train a very big neural network, then success is guaranteed!”\n- Ilya Sutskever\nFor years, scaling laws have been a predictable North Star for AI research. In fact, the success of early frontier labs like OpenAI has even been credited to their\nreligious level of belief\nin scaling laws. However, the continuation of scaling has recently been called into question by reports\n1\nclaiming that top research labs are struggling to create the next generation of better LLMs. These claims might lead us to wonder:\nWill scaling hit a wall and, if so, are there other paths forward?\nThis overview will answer these questions from the ground up, beginning with an in-depth explanation of LLM scaling laws and the surrounding research. The idea of a scaling law is simple, but there are a variety of public misconceptions around scaling—\nthe science behind this research is actually very specific\n. Using this detailed understanding of scaling, we will then discuss recent trends in LLM research and contributing factors to the “plateau” of scaling laws. Finally, we will use this information to more clearly illustrate the future of AI research, focusing on a few key ideas—\nincluding scaling\n—that could continue to drive progress.\nFundamental Scaling Concepts for LLMs\nTo understand the state of scaling for LLMs, we first need to build a general understanding of scaling laws. We will build this understanding from the ground up, starting with the concept of a power law. Then, we will explore how power laws have been applied in LLM research to derive the scaling laws we use today.\nWhat is a power law?\nPower laws\nare the fundamental concept that underlie LLM scaling. Put simply, power laws just describe a relationship between two quantities. For LLMs, the first of these quantities is the LLM’s test loss—\nor some other related performance metric (e.g., downstream task accuracy [7])\n—and the other is some setting that we are trying to scale, such as the number of model parameters. For example, we may see a statement like the following when studying the scaling properties of an LLM.\n“With enough training data, scaling of validation loss should be approximately a smooth power law as a function of model size.”\n- from [4]\nSuch a statement tells us that a measurable relationship exists between the model’s test loss and the total number of model parameters. A change to one of these quantities will produce a relative,\nscale-invariant\nchange in the other. In other words, we know from this relationship that increasing the total number of model parameters—\nassuming other conditions (e.g., having sufficient training data) are met\n—will result in a decrease of the test loss by predictable factor.\nPower law formulation.\nA basic power law is expressed via the equation below.\nThe two quantities being studied here are\nx\nand\ny\n, while\na\nand\np\nare constants that describe the relationship between these quantities. If we plot this power law function\n2\n, we get the figure shown below. We provide plots in both normal and log scale because most papers that study LLM scaling use a log scale.\nPlot of a basic power law between\nx\nand\ny\nHowever, the plots provided for LLM scaling do not look like the plot shown above—\nthey are usually flipped upside down\n; see below for an example.\n(from [1])\nThis is just an\ninverse\npower law, which can be formulated as shown below.\nThe equation for an inverse power law is nearly identically to that of a standard power law, but we use a negative exponent for\np\n. Making the exponent of the power law negative flips the plots upside down; see below for an example.\nPlot of an inverse power law between\nx\nand\ny\nThis inverse power law, when plotted using a log scale, yields the signature linear relationship that is characteristic of most LLM scaling laws. Nearly every paper covered in this overview will produce such a plot to study how scaling up a variety of different factors (e.g., size, compute, data, etc.) impacts an LLM’s performance. Now, let’s take a more practical look at power laws by learning about one of the first papers to study them in the context of LLM scaling [1].\nScaling Laws for Neural Language Models\n[1]\nIn the early days of language models, we did not yet understand the impact of scale on performance. Language models were a promising area of research, but the current generation of models (e.g., the\noriginal GPT\n) at that time had limited capabilities. We had yet to discover the power of larger models, and the path towards creating better language models was not immediately clear.\nIs the shape of the model (i.e., the number and size of layers) important? Does making a model larger help it to perform better? How much data is needed to train these larger models?\n“The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.”\n- from [1]\nIn [1], authors aim to answer these questions by analyzing the impact of several factors—\nsuch as model size, model shape, dataset size, training compute, and batch size\n—on model performance. From this analysis, we learn that LLM performance improves smoothly as we increase:\nThe number of model parameters.\nThe size of the dataset.\nThe amount of compute used for training.\nMore specifically,\na power law relationship is observed between each of these factors and the LLM’s test loss when performance is not bottlenecked by the other two factors.\nExperimental setup.\nTo fit their power laws, authors pretrain LLMs with sizes up to 1.5B parameters over subsets of the\nWebText2 corpus\n, containing anywhere from 22M to 23B tokens. All models are trained using a fixed context length of 1,024 tokens and a standard\nnext token prediction\n(\ncross-entropy\n) loss. The same loss is measured over a hold-out test set and used as our primary performance metric.\nThis setup matches the standard pretraining setup for most LLMs\n.\n(from [1])\nPower laws for LLM scaling.\nThe performance of LLMs trained in [1]—\nin terms of their test loss on WebText2\n—is shown to steadily improve with more parameters, data, and compute\n3\n. These trends span eight orders of magnitude in compute, six orders of magnitude in model size, and two orders of magnitude in dataset size.  The exact power law relationship and the equations that are fit to each of them are provided in the figure above. Each of the equations here are very similar to the inverse power law equation that we saw before. However, we set\na = 1\nand add an additional multiplicative constant inside of the parenthesis\n4\n.\nAuthors in [1] note one small detail that is necessary to properly fit these power laws. We do not include\npositional or token embeddings\nwhen counting the total number of model parameters, which yields cleaner scaling trends; see below.\n(from [1])\nThese power laws are only applicable if training is not bottlenecked by the other factors. So, all three of these components—\nmodel size, data, and compute\n—should be scaled up simultaneously for optimal performance. If we scale up any of these components in isolation, we will reach a point of diminishing returns.\nWhat do power laws tell us?\nAlthough the power law plots provided in [1] look quite promising, we should notice that these plots are generated using a log scale. If we generate normal plots (i.e., without log scale), we get the figures below, where we see that the shape of the power law resembles an exponential decay.\nPower law plots without log scale\nSuch a finding may seem counterintuitive given a lot of the online rhetoric around scaling and AGI. In many cases, it seems like the intuition we are being fed is that the quality LLMs improves exponentially with logarithmic increases in compute, but this is not the case whatsoever. In reality,\nincreasing the quality of an LLM becomes exponentially more difficult with scale\n.\n(from [1])\nOther useful findings.\nBeyond the power laws observed in [1], we also see that the other factors that are considered, such as model shape or architecture settings, have minimal influence on model performance; see above. Scale is by far the largest contributing factor for creating better LLMs—\nmore data, compute, and model parameters yields a smooth improvement in the LLM’s performance\n.\n“Larger models are significantly more sampleefficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.”\n- from [1]\nInterestingly, the empirical analysis in [1] indicates that larger LLMs tend to be more sample efficient, meaning that they reach the same level of test loss with less data relative to smaller models. For this reason,\npretraining an LLM to convergence is (arguably) sub-optimal\n. Instead, we could train a much larger model on less data, stopping the training process far before convergence. Such an approach is optimal in terms of the amount of training compute used but it does not account for inference costs. Practically speaking, we usually train smaller models over more data because smaller models are cheaper to host.\nAuthors also extensively analyze the relationship between model size and the amount of data used for pretraining, finding that the size of the dataset does not need to be increased as quickly as the model size.\nAn ~8X increase in model size requires a ~5X increase in the amount of training data to avoid overfitting\n.\n(from [1])\nThe scaling laws discovered in [1] are also replicated on several other datasets, where we see that the same scaling laws hold after adding a fixed offset to the test loss (i.e., to account for the fact that the dataset is different); see above. These results make a compelling case for LLM scaling. We achieve very clear and measurable benefits from training larger models longer and on more data, which created an appetite for pretraining LLMs at a much larger scale.\n“These results show that language modeling performance improves smoothly and predictably as we appropriately scale up model size, data, and compute. We expect that larger language models will perform better and be more sample efficient than current models.”\n- from [1]\nPractical Usage of Scaling Laws\nThe fact that large-scale pretraining is so beneficial presents us with a slight dilemma. The best results are achieved by training massive models over copious amounts of data. However, these training runs are\nincredibly expensive\n,\nwhich means they also incur a lot of risk\n. What if we spend $10 million training a model that fails to meet our expectations? Given the expense of pretraining, we cannot perform any model-specific tuning, and we must be sure that the model we train will perform well. We need to develop a strategy for tuning these models and forecasting their performance without spending too much money.\n(from [11])\nThis is where scaling laws come in. So far, we have seen some empirical analysis that was conducted to prove that scaling laws exist,\nbut these scaling laws also have a very practical use case within AI research\n. In particular, we can:\nTrain a bunch of smaller models using various training settings.\nFit scaling laws based on the performance of smaller models.\nUse the scaling law to extrapolate the performance of a much larger model.\nOf course, this approach has limitations. Predicting the performance of larger models from smaller models is difficult and can be inaccurate. Models may behave differently depending on scale. However,\nvarious approaches\nhave been proposed to make this more feasible, and scaling laws are now commonly used for this purpose. The ability to forecast the performance of larger models using scaling laws gives us more confidence (and peace of mind) as researchers. Plus, scaling laws provide a simple way to justify investment into AI research.\nScaling and The Age of Pretraining\n“This is what has been the driver of all progress we see today—extraordinarily large neural networks trained on huge datasets.”\n- Ilya Sutskever\nThe discovery of scaling laws catalyzed much of the recent progress in LLM research. To get better results, we just train increasingly large models on bigger (and\nbetter\n!) datasets. This strategy was followed to create several models within the GPT lineage, as well as most of the notable models from groups other than OpenAI. Here, we will take a deeper look at this progression of scaling research—\nrecently described by Ilya Sutskever as the “age of pretraining”\n5\n.\nThe GPT Lineage:\nGPT\n[2],\nGPT-2\n[3],\nGPT-3\n[4], and\nGPT-4\n[5]\nThe most widely known and visible application of LLM scaling laws was in the creation of OpenAI’s GPT lineage of models. We will primarily focus open earlier models in this lineage—\nup to GPT-3\n—because:\nThe details of these models were shared more openly.\nLater models heavily benefitted from advancements in\npost-training research\nin addition to scaling up the pretraining process.\nWe will also cover some of the known scaling results from models like GPT-4.\n(from [2])\nThe original GPT model\n[2] was actually quite small—\n12 layers and 117M parameters in total\n. This model is first pretrained over\nBooksCorpus\n, a dataset that contains the raw text of ~7,000 books. Then, we finetune the model to solve a variety of different downstream tasks by using a supervised training objective and creating a separate classification head for each task; see above. This paper was one of the first to perform large-scale,\nself-supervised pretraining\nof a\ndecoder-only transformer\n, which led to a few interesting findings:\nSelf-supervised pretraining on flat text is incredibly effective.\nUsing long, contiguous spans of text for pretraining is important.\nWhen pretrained in this manner, a single model can be finetuned to solve a wide variety of different tasks with state-of-the-art accuracy\n6\n.\nOverall, GPT was not an especially noteworthy model, but it laid some important foundations (i.e., decoder-only transformer and self-supervised pretraining) for later work that explored similar models at a much larger scale.\n(from [3])\nGPT-2\n[3] was proposed shortly after GPT and includes a collection of several models with sizes up to 1.5B parameters; see above. These models share the same architecture as the GPT model and are pretrained using the same self-supervised language modeling objective. However, GPT-2 makes two big changes to the pretraining process compared to GPT:\nThe models are pretrained on\nWebText\n, which is\ni)\nmuch larger than BooksCorpus and\nii)\ncreated by scraping data from the internet.\nThe models are not finetuned on downstream tasks. Instead, we solve tasks by performing\nzero-shot inference\n7\nwith the pretrained model.\nGPT-2 models fall short of state-of-the-art performance on most benchmarks\n8\n, but their performance consistently improves with the size of the model—\nscaling up the number of model parameters yields a clear benefit\n; see below.\n(from [3])\nAuthors in [3] also reveal that GPT-2 models—\ndespite their impressive results\n—still seem to underfit the WebText corpus. From this finding, we can infer that continued scaling of LLM pretraining—\nin terms of both model and data size\n—should be beneficial. Although the GPT-2 models were not particularly powerful, the analysis presented by these models provided the confidence we needed to continue scaling and eventually reach an inflection point in AI research.\n“A language model with sufficient capacity will begin to learn to infer and perform the tasks demonstrated in natural language sequences in order to better predict them, regardless of their method of procurement.”\n- from [3]\nGPT-3\n[4] was a watershed moment in AI research that definitively confirmed the benefit of large-scale pretraining for LLMs. The model has over 175B parameters, making it over 100x larger than the largest GPT-2 model; see below.\n(from [4])\nAgain, GPT-3 uses a decoder-only model architecture that is quite similar to that of prior models, but we pretrain the model over a much larger dataset based upon\nCommonCrawl\n. This dataset is ~10x bigger than the prior WebText dataset, and authors in [4] combine the larger pretraining dataset with several other sources of pretraining data, creating a mixture of different corpora; see below.\n(from [4])\nGPT-3 is primarily evaluated in [4] by using a\nfew-shot learning\napproach. The differences between few-shot prompting (used by GPT-3), zero-shot prompting (used by GPT-2), and finetuning (used by GPT) are illustrated below.\n(from [4])\nFew-shot learning is a new paradigm in which the LLM learns how to perform a task based upon examples that are placed within its context window. Authors in [4] refer to this concept as “in-context learning”. In this case, the LLM does not actually “learn”—\nthe model’s weights are not updated at all\n. Rather, the examples in the model’s input are used as context for generating a more accurate output. We see in [4] that GPT-3 is a highly capable few-shot learner, seeming to indicate that in-context learning is an emergent\n9\nability of larger models; see below.\n(from [4])\nWhen GPT-3 is evaluated on various language understanding tasks, we see that using a larger model significantly benefits few-shot learning performance, as shown in the figures below. Larger models make better—\nand more efficient\n—use of the information in their context windows relative to smaller models. GPT-3 is able to surpass state-of-the-art performance on several tasks via few-shot learning, and the model’s performance improves smoothly with size.\n(from [4])\nThe fact that a single model was able to perform this well across so many tasks was incredibly impressive at the time. Solving each of these tasks did not require any finetuning or changes to the underlying model—\nwe just had to tweak the model’s prompt\n. GPT-3 was one of the first true\nfoundation models\nto be released. This model ushered the next era of AI research and introduced a completely new and intuitive paradigm for interacting with LLMs (i.e., prompting).\nBeyond GPT-3.\nThe impressive performance of GPT-3 created an explosion of interest in LLM research, primarily focused upon large-scale pretraining. The next few models released by OpenAI—\nInstructGPT [8],\nChatGPT\nand GPT-4 [5]\n—used a combination of large-scale pretraining and new post-training techniques (i.e.,\nsupervised finetuning\nand\nreinforcement learning from human feedback\n) to drastically improve LLM quality. These models were so impressive that they even caused the amount of public interest in AI research to skyrocket.\n“GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior.”\n- from [5]\nAt this time, OpenAI began to publish fewer details on their research. Instead, new models were just released via\ntheir API\n, which kept the public from learning how these models were created. Luckily, some useful information can be gleaned from the material that OpenAI did publish. For example, InstructGPT [8]—\nthe predecessor to ChatGPT\n—has an associated paper that documents the model’s post training strategy in detail; see below. Given that this paper also states that GPT-3 is the base model for InstructGPT, we can reasonably infer that this model’s gains in performance are mostly unrelated to scaling up the pretraining process.\n(from [8])\nCompared to ChatGPT, GPT-4 had a noticeable improvement in capabilities. However, researchers chose to share very few of GPT-4’s technical details. The technical report for GPT-4 [5] simply tells us that:\nGPT-4 is transformer-based.\nThe model is pretrained using next token prediction.\nBoth public and licensed 3rd party data is used.\nThe model is finetuned with reinforcement learning from human feedback.\nNonetheless, the importance of scaling is very evident within this technical report. Authors note that a key challenge within this work was developing a scalable training architecture that behaves predictably across different scales, allowing the results of smaller runs to be extrapolated to provide confidence in the larger-scale (and significantly more expensive!) training exercises.\n“The final loss of properly-trained large language models is … approximated by power laws in the amount of compute used to train the model.”\n- from [5]\nLarge-scale pretraining is incredibly expensive, so we usually only get one chance at getting it right—\nthere is no room for model-specific tuning\n. Scaling laws play a key role in this process. We can train models using 1,000-10,000x less compute and use the results of these training runs to fit power laws. Then, these power laws can be used to predict the performance of much larger models. In particular, we see in [8] that the performance of GPT-4 is predicted using a power law that measures the relationship between compute and test loss; see below.\nScaling law formulation for training GPT-4 (from [5])\nThis expression looks nearly identical to what we have seen before, but it has an added irreducible loss term to account for the fact that the LLM’s test loss might never reach zero. Once fitted, the scaling law was used to predict the final performance of GPT-4 with very high accuracy; see below for a depiction. Here, we should note that the plot is not generated using a log scale, and we see that the improvement in loss clearly begins to decay with increasing compute!\n(from [5])\nAuthors in [5] also note that the test loss is not an easily interpretable metric and try to predict a variety of other performance metrics. For example, a scaling law is fit to predict the LLM’s pass rate on the\nHumanEval\ncoding benchmark. First, the problems in HumanEval are split into buckets based on their difficulty. Then, a scaling law is fit to predict the LLM’s pass rate. We see in [5] that the pass rate of GPT-4 can be accurately predicted on HumanEval using this approach based on experiments that require 1,000x less compute; see below.\n(from [5])\nAs we can see, scaling up the pretraining process is valuable. However, large-scale pretraining is also very expensive.\nScaling laws make this process more predictable\n, allowing us to avoid needless or excessive compute costs.\nChinchilla:\nTraining Compute-Optimal Large Language Models\n[5]\n(from [9])\nIn [1], authors recommend increasing model size faster than the size of the dataset when scaling up LLM pretraining. However, most of the pretraining research after GPT-3 indicated that we should do the opposite. We trained models that were significantly larger than GPT-3—\nsuch as the 530B parameter MT-NLG [9] model\n—but the size of the dataset used to train these models was similar to that of GPT-3; see above. These models did not improve upon the performance of GPT-3, whereas models that used a combination of more parameters with more data (e.g., Gopher [10]) performed much better; see below.\n(from [10])\nCompute-optimal scaling laws.\nInspired by these observations, authors in [6] completely reconsidered the best practices for scaling laws that were originally proposed in [1]. The analysis of scaling laws in [6] was conducted with much larger models, yielding results that are slightly different from before. More specifically, LLMs with sizes ranging from 70M to 17B parameters are trained over datasets with sizes exceeding one trillion tokens; see below.\n(from [10])\nBy training LLMs with many different combinations of model and data size, we can discover a power law that predicts an LLM’s test loss as a function of these factors. From these power laws, we can determine which training settings work best for a given compute budget. Authors in [6] argue that compute-optimal training\n10\nshould scale model and data size proportionally. Such a finding reveals that most LLM are undertrained for their size—\nwe would benefit from training existing LLMs over significantly more data\n. For example, the scaling laws fit in [6] predict that Gopher should have been trained on a 20x larger dataset!\n“The amount of training data that is projected to be needed is far beyond what is currently used to train large models.”\n- from [6]\nChinchilla.\nThe analysis provided in [6] emphasizes the importance of data scale.\nLarge models need to be trained over more data to reach their best performance\n. To validate this finding, authors train a 70 billion parameter LLM, called Chinchilla. Compared to prior models, Chinchilla is smaller but has a larger pretraining dataset—\n1.4T training tokens in total\n. Chinchilla uses the same data and evaluation strategy as Gopher [10]. Despite being 4x smaller than Gopher, Chinchilla consistently outperforms the larger model; see below.\n(from [6])\nScaling laws proposed by Chinchilla [6] became a standard in AI research for years afterward.\n“Chinchilla-optimal”\nis now a commonly-used term. Even today, after a wide variety of additional scaling research has been published, Chinchilla and its associated scaling laws are referenced constantly.\nThe “Death” of Scaling Laws\nScaling laws have recently become a popular (and contentious) topic within AI research. As we have seen throughout this overview, scaling has fueled most improvements in AI throughout the age of pretraining. As the pace of model releases and improvements slowed in the latter half of 2024\n11\n, however, we began to see widespread questioning of model scaling, seeming to indicate that AI research—\nand scaling laws in particular\n—could be hitting a wall.\nReuters\nclaims that OpenAI is shifting their product strategy due to hitting a plateau in scaling current methods.\nThe Information\nclaims that the rate of improvement in GPT models is starting to slow down.\nBloomberg\nhighlights the difficulties being faced by several frontier labs in trying to build AI that is more capable.\nTechCrunch\nclaims that scaling is starting to yield diminishing returns.\nTime\npublishes a nuanced essay that highlights various contributing factors to the narrative that AI research is slowing down.\nIlya Sutskever states that\n“pretraining as we know it will end”\nduring his\ntest of time award speech\nat NeurIPS’24.\nAt the same time, many experts are arguing the opposite. For example,\nDario Amodei\n(Anthropic CEO) has said that scaling is\n“probably… going to continue”\n, while Sam Altman has continued pushing the narrative that\n“there is no wall”\n. In this section, we will add more color to this discussion by providing a grounded explanation of the current state of scaling and various issues that may exist.\nSlower Scaling: What does it mean? Why is it happening?\n“Both narratives can be true: Scaling is still working at a technical level. The rate of improvement for users is slowing.”\n-\nNathan Lambert\nSo…\nis scaling slowing down?\nThe answer is complex and highly dependent upon our exact definition of “slowing down”. So far, the most reasonable response that I’ve seen to this question is that both answers are correct. For this reason, we will not try to answer this question. Instead, we will simply take a deeper look at what exactly the research tells us about this topic so that we can build a more nuanced understanding of the current (and future) state of scaling for LLMs.\nWhat do scaling laws tell us?\nFirst, we need to recall the technical definition of a scaling law. Scaling laws define a relationship, based upon a power law, between training compute (or model / dataset size) and the test loss of an LLM. However,\nthe nature of this relationship is often misunderstood\n. The idea of getting exponential performance improvements from logarithmic increases in compute is a myth. Scaling laws look more like an exponential decay, meaning that we will have to work harder over time to get further performance improvements; see below.\n(from [5])\nIn other words,\nscaling laws plateau naturally over time\n. In this way, the “slowdown” we are currently experiencing is arguably an expected part of LLM scaling laws.\n“Practitioners often use downstream benchmark accuracy as a proxy for model quality and not loss on perplexity evaluation sets.”\n- from [7]\nDefining performance.\nHow do we measure whether LLMs are improving or not?\nFrom the perspective of scaling laws, LLM performance is usually measured via the model’s test loss during pretraining, but the impact of lower test loss on an LLM’s capabilities is unclear.\nDoes lower loss lead to higher accuracy on downstream tasks? Does lower loss cause an LLM to acquire new capabilities?\nThere is a disconnect here between what scaling laws tell us and what we actually care about:\nScaling laws tell us that increasing the scale of pretraining will smoothly decrease the LLM’s test loss.\nWe care about getting a “better” LLM.\nDepending on who you are, your expectations of new AI systems—\nand the approach that you use to evaluate these new systems\n—will vary drastically. Average AI users tend to focus upon general chat applications, while practitioners often care about an LLM’s performance on downstream tasks. In contrast, researchers at top frontier labs seem to have high (and very particular) expectations of AI systems; e.g.,\nwriting a Ph.D. thesis\nor solving\nadvanced mathematical reasoning problems\n. Given that LLMs have such broad capabilities, evaluation is difficult, and there are many lenses from which we can view an LLM’s performance; see below.\n(from [15])\nGiven this drastic variance in model expectations, providing definitive proof that scaling is “working” will always be a struggle. We need a more specific definition of success for scaling laws. If the science tells us that larger models will achieve lower loss, this does not mean that new models will meet everyone’s expectations. A failure to achieve AGI or surpass the capabilities of award-winning human mathematicians is not proof that scaling is not still working at a technical level! Put differently, one could argue that the “slowdown” of scaling is a perception and expectation problem rather than a technical problem with scaling laws.\nData death.\nTo scale up LLM pretraining, we must increase both model and dataset size. Earlier research [1] seemed to indicate that the amount of data was not as important as the size of the model, but we see with Chinchilla [6] that dataset size is equally important. Plus, more recent work argues that most researchers prefer to “overtrain” their models—\nor pretrain them on datasets that go beyond Chinchilla-optimality in size\n—to save on inference costs [7].\n“Scaling studies usually focus on the compute-optimal training regime... As larger models are more expensive at inference, it is now common practice to over-train smaller models.”\n- from [7]\nAll of this research brings us to one simple conclusion—\nscaling up LLM pretraining will require that we create larger pretraining datasets\n. This fact forms the basis for one of the key criticisms of LLM scaling laws. Many researchers believe that there may not be enough data available to continue scaling the pretraining process. For context, a large majority of pretraining data used for current LLMs is obtained via web scraping; see below. Given that we only have one internet, finding completely new sources of large-scale, high-quality pretraining data may be hard.\n(\nsource\n)\nEven Ilya Sutskever has\nrecently made this argument\n, claiming that\ni)\ncompute is growing quickly but\nii)\ndata is not growing due to reliance upon web scraping. Therefore, he believes that we cannot continue scaling up the pretraining process forever. Pretraining as we know it will end, and we must find new avenues of progress for AI research. In other words,\n“we have achieved peak data”\n.\nThe Next Generation of Scale for Pretraining\nScaling will eventually lead to diminishing returns, and the data-centric argument against the continuation of scaling is both sound and compelling. However, there are still several directions of research that could improve the pretraining process.\nSynthetic data.\nTo scale up the pretraining process several orders of magnitude, we will likely need to rely upon synthetically-generated data. Despite worries that over reliance upon synthetic data will lead to diversity issues [14], we are seeing increased—\nand\nseemingly successful\n—usage of synthetic data for LLMs [12]. Additionally, curriculum learning [13] and\ncontinued pretraining\nstrategies have led to a variety of meaningful improvements via adjustments to pretraining data; e.g., changing data mixtures or adding instruction data at the end of pretraining.\n(from [7])\nPractical scaling laws.\nRecent research has attempted to address the limitations of test loss-based scaling laws. For example, authors in [7] define scaling laws that can be used to predict the performance of an LLM on downstream benchmarks from the\nLLM Foundry\n; see above. Interpreting these kinds of metrics is much easier for humans. We might not know what a 5% decrease in test loss means, but jumping from 85% to 90% accuracy on our benchmark of interest is usually easy to grasp. Several other works have also explored the idea of using scaling laws to provide more practical and meaningful estimates of LLM performance; e.g., after post training and quantization [16] or during the pretraining process [17].\nDeepSeek-v3.\nDespite recent arguments, we are still seeing semi-frequent advancements by scaling the LLM pretraining process. For example, DeepSeek-v3 [18]—\na 671B parameter\n12\nmixture-of-experts\n(MoE) model\n—was recently released. In addition to being open-source, this model is pretrained on 14.8T tokens of text and surpasses the performance of GPT-4o and Claude-3.5-Sonnet; see below for the model’s performance and\nhere\nfor the license. For reference, LLaMA-3 models are trained on over 15T of raw text data; see\nhere\nfor more details.\n(from [18])\nThe ability to outperform models like GPT-4o is a significant jump for open weights LLMs—\neven the\nlargest LLaMA models\nhave fallen short of this goal\n. DeepSeek-v3 adopts a variety of interesting tricks:\nAn optimized MoE architecture from\nDeepSeek-v2\n.\nA new auxiliary-loss-free strategy for\nload balancing\nthe MoE.\nA\nmulti-token prediction\ntraining objective.\nDistillation of reasoning capabilities from a long-chain-of-thought model (i.e., similar to\nOpenAI’s o1\n).\nThe model also undergoes post-training, including supervised finetuning and reinforcement learning from human feedback, to align it to human preferences.\n“We train DeepSeek-V3 on 14.8T high-quality and diverse tokens. The pre-training process is remarkably stable. Throughout the entire training process, we did not encounter any irrecoverable loss spikes or have to roll back.”\n- from [8]\nHowever, the biggest key to DeepSeek-v3’s impressive performance is pretraining scale—\nthis is a massive model trained over an equally-massive dataset\n! Training such a large model is difficult for a variety of reasons (e.g., GPU failures and loss spikes). DeepSeek-v3 has a surprisingly stable pretraining process and is trained at a reasonable cost by LLM standards; see below.\nThese results indicate that larger-scale pretraining jobs are becoming more manageable and efficient over time\n.\n(from [18])\nIncreasing scale by an OOM.\nTo continue testing our scaling laws, we must train LLMs that are several orders of magnitude beyond current models. Setting aside our opinions on the utility of scaling, there are still a variety of limitations that stand in the way of training models of this scale. We will need:\nLarger compute clusters\n13\n.\nMore (and better) hardware.\nMassive amounts of power.\nNew algorithms (e.g., for larger-scale distributed training,\npotentially spanning multiple data centers\n).\nTraining the next generations of models is not just a matter of securing funding for more GPUs, it is a multi-disciplinary feat of engineering. Such complex efforts take time. For reference, GPT-4 was released in March of 2023, nearly three years—\n33 months in particular\n—after the release of GPT-3. It is reasonable to expect a similar timeline (if not longer) for unlocking another 10-100x increase in scale.\n“At every order of magnitude scale up, different innovations have to be found.”\n-\nEge Erdil\n(\nEpoch AI\n)\nThe Future(s) of AI Research\nNow that we more deeply understand the state of scaling for pretraining, let’s assume (purely for the purpose of discussion) that pretraining research will hit a sudden wall. Even if model capabilities do not improve at all in the near future, there are a variety of ways in which AI research can continue to rapidly progress. We have already talked about some of these topics (e.g., synthetic data). In this section, we will focus on two topics in particular that are currently popular:\nLLM systems / agents.\nReasoning models.\nBuilding Useful LLM Systems\nMost LLM-based applications today operate in a single-model paradigm. In other words, we solve tasks by passing that task to a single LLM and directly using the model’s output as the answer to that task; see below.\nIf we want to improve such a system (i.e., solve harder tasks with higher accuracy), we could simply improve the capabilities of the underlying model, but such an approach is dependent upon the creation of more capable models. Instead, we could go beyond the single-model paradigm by building an LLM-based system that combines several LLMs—\nor other components\n—to solve a complex task.\nLLM system basics.\nThe goal of an LLM system is to break complex tasks into smaller parts that are easier to solve for an LLM or other module. There are two primary strategies we can use to accomplish this goal (depicted above):\nTask decomposition\n: break the task itself into smaller sub-tasks that can be solved individually and aggregated\n14\nafterwards to form a final answer.\nChaining\n: solve a task or sub-task by making several sequential calls to an LLM instead of a single call.\nThese strategies can be used alone or in tandem. For example, assume that we want to build a system to summarize books. To do this, we could decompose the task by first summarizing each chapter of the book. From here, we can either:\nFurther decompose the task into smaller chunks of text to summarize (i.e., similar to\nrecursive / hierarchical decomposition\n).\nChain several LLM calls together; e.g., have one LLM extract all important facts or information from the chapter and another LLM produce the chapter summary based on these key facts.\nThen, we can aggregate these results by asking an LLM to summarize the concatenated chapter summaries, forming a summary of the full novel. Most complex tasks can be decomposed into simple parts that are easy to solve, which makes such LLM systems very powerful. These systems can become incredibly sophisticated as we perform more extensive decomposition and chaining, making them an interesting (and impactful) area of applied AI research.\nBuilding LLM-based products.\nDespite the success and popularity of LLMs, the number of practical (and widely-adopted) use cases for LLMs is still very small. The largest use cases for LLMs today are code generation and chat, both of which are relatively obvious applications for LLMs\n15\n; see below.\n(\nsource\n)\nGiven that there are so many ripe domains for the application of LLMs,\nsimply building more LLM-based products that are genuinely useful is an important area of applied AI research\n. We already have access to very capable models, but using these models to build a product worth using is an entirely different problem. Solving this problem requires learning how to build reliable and capable LLM systems.\n(from [19])\nAgents.\nThe line between LLM systems and agents is blurry because the term “agent” has been\noverloaded\nby the AI community. However, the key concept that we should understand is that\nLLM systems can be expanded in a variety or interesting and meaningful ways\n. For example, we can augment LLMs by teaching them how to use tools (e.g., calculators, search engines, etc.) when solving a problem; see above. Additionally, we can allow LLMs to execute their own programs or even perform actions for us; e.g., booking a hotel or sending an email. The many modules and tools that can be integrated with an LLM present endless possibilities for building LLM systems that are more capable and useful.\nRobustness\nis one of the biggest roadblocks to building more capable LLM / agent systems. Assume that we have an LLM system that makes ten different calls to an LLM. Additionally, let’s assume that each of these LLM calls have a 95% likelihood of success and that all of the calls need to succeed in order to generate the correct final outputs. Although the individual components of this system are reasonably accurate,\nthe overall system has a success rate of 60%\n!\n(from [20])\nAs we add more components, this problem becomes exponentially worse, which limits the complexity of LLM / agent systems that we can build. Building more complex systems will require that we drastically improve the robustness of each individual system component.\nRecent research\nindicates that robustness can be improved via scaling. However, we can also improve robustness via better meta-generation algorithms; see above. Instead of generating a single output from an LLM, these algorithms use ideas like parallel decoding, (step-level) verification, critiques, and more to obtain a more refined and accurate output from an LLM.\n(from [20])\nThis area of research is rapidly progressing and is likely to become a key driver of progress in AI research; see [20] for an in-depth survey of the topic. As meta-generation algorithms improve, LLMs will become more robust, and we will become capable of building increasingly complex LLM / agent systems.\nReasoning Models and New Scaling Paradigms\nA common criticism of early LLMs was that they simply memorized data and had very little reasoning capabilities. However, the inability of LLMs to reason has been largely debunked over the last few years. We have learned from recent research that these models likely had the inherent ability to reason all along, but we have to use the correct prompting or training approach to elicit this ability.\nChain of thought (CoT) prompting\n[22] was one of the first techniques to demonstrate the reasoning capabilities of LLMs. The approach is simple and prompt-based. We just ask the LLM to provide an explanation for its response prior to generating the actual response; see\nhere\nfor more details. When an LLM generates a rationale that outlines the\nstep-by-step\nprocess used to arrive at a response, its reasoning capabilities improve significantly. Plus, this explanation is human-readable and can make the model’s output more interpretable!\n(from [22])\nThe idea of a chain of thought is both generic and powerful. In fact, chains of thought have become the key concept behind improving LLM reasoning capabilities, and we have seen this technique repurposed in many ways:\nLLM-as-a-Judge-style evaluation models usually provide scoring rationales prior to generating a final evaluation result [23, 24].\nSupervised finetuning and instruction tuning strategies have been proposed for teaching smaller / open LLMs to write better chains of thought [25, 26].\nLLMs are commonly asked to reflect upon and critique or verify their own outputs, then revise their output based on this information [12, 27].\nComplex reasoning is an active research topic that is quickly progressing. New training algorithms that teach LLMs to incorporate (step-level) verification [28, 29] into their reasoning process have shown promising results, and we will likely continue to see improvements as new and better training strategies emerge.\nOpenAI’s o1 reasoning model\n[21] marks a significant jump in the reasoning capabilities of LLMs. The reasoning strategy used by o1 is heavily based upon chains of thought. Similar to how humans think before responding to a question, o1 will spend time “thinking” before providing a response. Practically speaking, the “thoughts” that o1 generates are just long chains of thought that the model uses to think through a problem, break the problem into simpler steps, try various approaches to solving the problem, and even correct its own mistakes\n16\n.\n“OpenAI o1 [is] a new large language model trained with RL to perform complex reasoning. o1 thinks before it answers—it can produce a long internal chain of thought before responding to the user.”\n- from [21]\nThe details of o1’s exact training strategy are not publicly shared. However, we know that o1 is taught to reason using a “large-scale reinforcement learning” algorithm that is “highly data-efficient” and focuses upon refining the model’s ability to generate useful chains of thought. Based upon\npublic comments\nfrom OpenAI researchers and\nrecent rhetoric\nregarding o1, the model seems to be trained using pure reinforcement learning, which contradicts earlier opinions that o1 may be using some form of tree search at inference time.\nComparison between GPT-4o and o1 on reasoning-heavy tasks (from [21])\nAs mentioned, the performance of o1 on complex reasoning tasks is impressive. o1 outperforms GPT-4o on nearly all reasoning-heavy tasks; see above. As an example of o1’s reasoning capabilities, the model:\nPlaces in the 89th percentile on competitive programming questions from\nCodeforces\n.\nReaches the top 500 students in the US in a qualifier for the USA Math Olympiad (\nAIME\n).\nExceeds the accuracy of human PhD candidates on graduate-level physics, biology and chemistry problems (\nGPQA\n).\n(from [22])\nFrom o1 to o3.\nOne of the most interesting aspects of o1 is that the model’s reasoning capabilities can be improved by using more compute at inference time. To solve problems of increasing complexity, the model can simply generate progressively longer chains of thought; see\nhere\nfor examples. Using more inference-time compute to generate these longer chains of thought yields a smooth increase in the model’s reasoning performance; see below.\n“We have found that the performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute).”\n- from [22]\nSimilarly, we see in the figure above that o1’s performance improves smoothly as we invest more compute into training via reinforcement learning. This is exactly the approach that was followed to create the o3 reasoning model. The evaluation results for this model were\npreviewed\nby OpenAI in late 2024, and very few details about o3 have been publicly shared. However, given that this model was released so quickly after o1 (i.e., three months later), o3 is most likely a\n“scaled up” version of o1\nthat invests more compute into reinforcement learning.\n(\nsource\n)\nThe o3 model has not yet been released at the time of writing, but the results achieved by scaling o1 are incredibly impressive (even shocking in some cases). The most notable achievements of o3 are listed below:\nA score of 87.5% on the\nARC-AGI benchmark\non which GPT-4o achieves only 5% accuracy. o3 is the first model to exceed human-level performance of 85% on ARC-AGI.  This benchmark has been described as a “North Star” towards AGI and has remained unbeaten\n17\nfor over five years.\nAn accuracy of 71.7% on\nSWE-Bench Verified\nand an\nElo score\nof 2727 on Codeforces, which ranks o3 among the top 200 human competitive programmers on the planet.\nAn accuracy of 25.2% on EpochAI’s\nFrontierMath benchmark\n,\nimproving upon the previous state-of-the-art accuracy of 2.0%\n. This benchmark was described by\nTerence Tao\nas “incredibly difficult” and likely to be unsolved by AI systems for “several years at least”.\nA distilled version of o3, called o3-mini, was also previewed, which performs very well and comes with significant improvements in compute efficiency.\n(from [21] and\nhere\n)\nNew paradigms for scaling.\nAfter reading this overview, many of the plots presented by o1 and o3 (see above) might look pretty familiar—\nthese are log-scale plots where we see a smooth, linear increase in performance with more compute\n! In other words, we see a clear power law relationship between the performance of these reasoning models and two different quantities:\nTraining-time (reinforcement learning) compute.\nInference-time compute.\nScaling o1-style models differs from traditional scaling laws. Instead of scaling up the pretraining process, we are scaling the amount of compute invested into post training and inference.\nThis is a completely new scaling paradigm\n, and the results achieved by scaling reasoning models are great so far. Such a finding shows us that other avenues of scaling—\nbeyond pretraining\n—clearly exist. With the advent of reasoning models, we have discovered our next hill to climb. Although it may come in a different form,\nscaling will continue to drive progress in AI research\n.\nClosing Thoughts\nWe now have a clearer view of scaling laws, their impact on LLMs, and the future directions of progress for AI research. As we have learned, there are many contributing factors to the recent criticism of scaling laws:\nThe natural decay in scaling laws.\nThe high variance in expectations of LLM capabilities.\nThe latency of large-scale, inter-disciplinary engineering efforts.\nThese issues are valid,\nbut\nnone of them indicate that scaling is not still working as expected\n. Investments into large-scale pretraining will (and should) continue, but improvements will become exponentially harder over time. As a result, alternative directions of progress (e.g., agents and reasoning) will become more important. As we invest into these new areas of research, however, the fundamental idea of scaling will continue to play a massive role. Whether scaling will continue is not a question.\nThe true question is what we will scale next\n.\nNew to the newsletter?\nHi! I’m\nCameron R. Wolfe\n, Deep Learning Ph.D. and Machine Learning Scientist at\nNetflix\n. This is the Deep (Learning) Focus newsletter, where I help readers better understand important topics in AI research. If you like the newsletter, please subscribe, share it, or follow me on\nX\nand\nLinkedIn\n!\nSubscribe\nBibliography\n[1] Kaplan, Jared, et al. \"Scaling laws for neural language models.\"\narXiv preprint arXiv:2001.08361\n(2020).\n[2] Radford, Alec. \"Improving language understanding by generative pre-training.\" (2018).\n[3] Radford, Alec, et al. \"Language models are unsupervised multitask learners.\"\nOpenAI blog\n1.8 (2019): 9.\n[4] Brown, Tom, et al. \"Language models are few-shot learners.\"\nAdvances in neural information processing systems\n33 (2020): 1877-1901.\n[5] Achiam, Josh, et al. \"Gpt-4 technical report.\"\narXiv preprint arXiv:2303.08774\n(2023).\n[6] Hoffmann, Jordan, et al. \"Training compute-optimal large language models.\"\narXiv preprint arXiv:2203.15556\n(2022).\n[7] Gadre, Samir Yitzhak, et al. \"Language models scale reliably with over-training and on downstream tasks.\"\narXiv preprint arXiv:2403.08540\n(2024).\n[8] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\"\nAdvances in neural information processing systems\n35 (2022): 27730-27744.\n[9] Smith, Shaden, et al. \"Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model.\"\narXiv preprint arXiv:2201.11990\n(2022).\n[10] Rae, Jack W., et al. \"Scaling language models: Methods, analysis & insights from training gopher.\"\narXiv preprint arXiv:2112.11446\n(2021).\n[11] Bhagia, Akshita, et al. \"Establishing Task Scaling Laws via Compute-Efficient Model Ladders.\"\narXiv preprint arXiv:2412.04403\n(2024).\n[12] Bai, Yuntao, et al. \"Constitutional ai: Harmlessness from ai feedback.\"\narXiv preprint arXiv:2212.08073\n(2022).\n[13] Blakeney, Cody, et al. \"Does your data spark joy? Performance gains from domain upsampling at the end of training.\"\narXiv preprint arXiv:2406.03476\n(2024).\n[14] Chen, Hao, et al. \"On the Diversity of Synthetic Data and its Impact on Training Large Language Models.\"\narXiv preprint arXiv:2410.15226\n(2024).\n[15] Guo, Zishan, et al. \"Evaluating large language models: A comprehensive survey.\"\narXiv preprint arXiv:2310.19736\n(2023).\n[16] Xu, Zifei, et al. \"Scaling laws for post-training quantized large language models.\"\narXiv preprint arXiv:2410.12119\n(2024).\n[17] Xiong, Yizhe, et al. \"Temporal scaling law for large language models.\"\narXiv preprint arXiv:2404.17785\n(2024).\n[18] DeepSeek-AI et al. \"DeepSeek-v3 Technical Report.\"\nhttps://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf\n(2024).\n[19] Schick, Timo, et al. \"Toolformer: Language models can teach themselves to use tools.\" arXiv preprint arXiv:2302.04761 (2023).\n[20] Welleck, Sean, et al. \"From decoding to meta-generation: Inference-time algorithms for large language models.\"\narXiv preprint arXiv:2406.16838\n(2024).\n[21] OpenAI et al. “Learning to Reason with LLMs.”\nhttps://openai.com/index/learning-to-reason-with-llms/\n(2024).\n[22] Wei, Jason, et al. \"Chain-of-thought prompting elicits reasoning in large language models.\"\nAdvances in neural information processing systems\n35 (2022): 24824-24837.\n[23] Liu, Yang, et al. \"G-eval: Nlg evaluation using gpt-4 with better human alignment.\"\narXiv preprint arXiv:2303.16634\n(2023).\n[24] Kim, Seungone, et al. \"Prometheus: Inducing fine-grained evaluation capability in language models.\"\nThe Twelfth International Conference on Learning Representations\n. 2023.\n[25] Ho, Namgyu, Laura Schmid, and Se-Young Yun. \"Large language models are reasoning teachers.\"\narXiv preprint arXiv:2212.10071\n(2022).\n[26] Kim, Seungone, et al. \"The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning.\"\narXiv preprint arXiv:2305.14045\n(2023).\n[27] Weng, Yixuan, et al. \"Large language models are better reasoners with self-verification.\"\narXiv preprint arXiv:2212.09561\n(2022).\n[28] Lightman, Hunter, et al. \"Let's verify step by step.\"\narXiv preprint arXiv:2305.20050\n(2023).\n[29] Zhang, Lunjun, et al. \"Generative verifiers: Reward modeling as next-token prediction.\"\narXiv preprint arXiv:2408.15240\n(2024).\n1\nThe two primary reports are from\nThe Information\nand\nReuters\n.\n2\nWe use the following settings to generate the plot:\na = 1\n,\np = 0.5\n, and\n0 < x < 1\n.\n3\nCompute is defined in [1] as\n6NBS\n, where\nN\nis the number of model parameters,\nB\nis the batch size used during training, and\nS\nis the total number of training steps.\n4\nThis extra multiplicative constant does not change the behavior of the power law. To understand why this is the case, we must understand the definition of\nscale invariance\n. Because power laws are scale invariant, the fundamental characteristics of a power law are the same even when we scale up or down by a certain factor. The behavior observed will be the same at any scale!\n5\nThis description was from Ilya’s test of time award for\nthis paper\nat NeurIPS’24.\n6\nAlthough this might seem obvious now, we should remember that most NLP tasks at the time (e.g., summarization and QA) had entire areas of research devoted to them! Each of these tasks had associated task-specific architectures that were specialized in performing that task, and GPT was a single generic model that could outperform most of these architectures across several different tasks.\n7\nThis means that we just describe each task in the LLM’s prompt and use the same model to solve different tasks—\nonly the prompt changes between tasks\n.\n8\nThis is to be expected because these models use zero-shot inference and are not finetuned at all on any of the downstream tasks.\n9\nBy an “emergent” ability, we mean a skill possessed by the LLM that only emerges after a certain scale has been reached (e.g., a sufficiently large model).\n10\nHere, we define “compute-optimal” as the training setting that yields the best possible performance—\nin terms of test loss\n—at a fixed training compute cost.\nbest performance at a fixed compute cost\n11\nFor example, Anthropic has continually pushed back their release of\nClaude 3.5 Opus\n, Google has only released the\nflash variant of Gemini-2\n, and OpenAI only released\nGPT-4o\nin 2024 (until\no1\nand o3 were released in December), which was arguably not significantly more capable than GPT-4.\n12\nOnly 37 billion of these parameters are active during inference for a single token.\n13\nFor example, xAI has recently built a\nnew datacenter\nwith 100,000 NVIDIA GPUs in Memphis, while Anthropic leadership has\nvoiced a desire\nto increase their compute spend by up to 100x over the next few years.\n14\nThe aggregation step can be implemented in a variety of ways. For example, we can manually aggregate responses (e.g., via concatenation), use an LLM, or pretty much anything in between!\n15\nThis is not because these tasks are simple. Both code generation and chat are hard to solve, but they are (arguably) fairly obvious applications for LLMs.\n16\nOpenAI has chosen to hide these long chains of thought from the users of o1. The argument behind this choice is that these rationales provide insight into the model’s though process that can be used to debug or monitor the model. However, the model should be allowed to express its pure thoughts without any of the safety filtering that is necessary for user-facing model outputs.\n17\nCurrently, ARC-AGI is technically still unbeaten because o3 exceeds the compute requirement for the benchmark. However, the model still achieves an accuracy of 75.7% using a lower compute setting.",
  "metadata": {
    "title": "Scaling Laws for LLMs: From GPT-3 to o3",
    "description": "Understanding the current state of LLM scaling and the future of AI research...",
    "keywords": null,
    "author": "Cameron R. Wolfe, Ph.D.",
    "og:type": "article",
    "og:title": "Scaling Laws for LLMs: From GPT-3 to o3",
    "og:description": "Understanding the current state of LLM scaling and the future of AI research...",
    "og:image": "https://substackcdn.com/image/fetch/$s_!p_mp!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F44e9a03a-b5c7-4eb2-aef8-ef019c38d671_2578x1440.png",
    "og:url": "https://cameronrwolfe.substack.com/p/llm-scaling-laws",
    "twitter:title": "Scaling Laws for LLMs: From GPT-3 to o3",
    "twitter:description": "Understanding the current state of LLM scaling and the future of AI research...",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!3cCQ!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fcameronrwolfe.substack.com%2Fapi%2Fv1%2Fpost_preview%2F152758713%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2025-01-06T15:51:52.921Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!3iKS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98592f1d-1d20-4c88-a681-9d4dac0289d4_2014x1130.png",
      "bytes": 134745,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/039.png",
      "src": "https://substackcdn.com/image/fetch/$s_!3iKS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98592f1d-1d20-4c88-a681-9d4dac0289d4_2014x1130.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!VKCD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4e231b5-3002-4bec-a733-512e3f6730a5_240x68.png",
      "bytes": 3462,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-b57e476e45b03dcd7a4d.png",
      "src": "https://substackcdn.com/image/fetch/$s_!VKCD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4e231b5-3002-4bec-a733-512e3f6730a5_240x68.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!pZed!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b410c6-03f9-4214-8d6a-074b1cbbf6ec_800x300.png",
      "bytes": 21505,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-963be4fe5a5f15345547.png",
      "src": "https://substackcdn.com/image/fetch/$s_!pZed!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b410c6-03f9-4214-8d6a-074b1cbbf6ec_800x300.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Av0X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82fbae3c-e5d1-4936-8328-057eba9893a3_806x524.png",
      "bytes": 37663,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-702f17fbf68b9ded3d4a.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Av0X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82fbae3c-e5d1-4936-8328-057eba9893a3_806x524.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!EmoD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19681045-c6e7-41dc-aee7-8fb87ec02b5f_376x178.png",
      "bytes": 9751,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-9b46678daa945c0f69ed.png",
      "src": "https://substackcdn.com/image/fetch/$s_!EmoD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19681045-c6e7-41dc-aee7-8fb87ec02b5f_376x178.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!ybvx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb36a47-1a10-41c1-ad10-dd0a5a7df7d2_800x300.png",
      "bytes": 19933,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-2ab530211f85c1262b2c.png",
      "src": "https://substackcdn.com/image/fetch/$s_!ybvx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb36a47-1a10-41c1-ad10-dd0a5a7df7d2_800x300.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!OvRU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F071a53ae-e1d5-4af6-bcd5-4402cc27e924_2144x1244.png",
      "bytes": 334938,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-f925967ddb5bc5af1c83.png",
      "src": "https://substackcdn.com/image/fetch/$s_!OvRU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F071a53ae-e1d5-4af6-bcd5-4402cc27e924_2144x1244.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!0b4b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe90d52a4-413f-4140-977d-3e0c52497dae_1968x808.png",
      "bytes": 240442,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-b48a790be4a08cb1b3e5.png",
      "src": "https://substackcdn.com/image/fetch/$s_!0b4b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe90d52a4-413f-4140-977d-3e0c52497dae_1968x808.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!L-l2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabf07f96-f9d9-467f-a459-fb24788f4e33_2540x630.png",
      "bytes": 101664,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-fbc0b1db284c76bc29bc.png",
      "src": "https://substackcdn.com/image/fetch/$s_!L-l2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabf07f96-f9d9-467f-a459-fb24788f4e33_2540x630.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!BX8S!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9603be2-d7fd-4f59-bda2-1e5f76f17f8d_1254x502.png",
      "bytes": 110921,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-d56060a5b7abf2ee7f02.png",
      "src": "https://substackcdn.com/image/fetch/$s_!BX8S!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9603be2-d7fd-4f59-bda2-1e5f76f17f8d_1254x502.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!ELFU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb29aeb01-107f-4b6f-b462-7477f6bd863b_1256x576.png",
      "bytes": 101171,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-8c67ac1c0c97d4eb15df.png",
      "src": "https://substackcdn.com/image/fetch/$s_!ELFU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb29aeb01-107f-4b6f-b462-7477f6bd863b_1256x576.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!aoTo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7bae506a-dddf-440b-be73-87bafe6d7d0f_1884x1032.png",
      "bytes": 187264,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-d62df28e2f0fbaea06a1.png",
      "src": "https://substackcdn.com/image/fetch/$s_!aoTo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7bae506a-dddf-440b-be73-87bafe6d7d0f_1884x1032.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!ie6t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F216100ce-7a2b-4313-b047-25451e29620f_1872x936.png",
      "bytes": 153003,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-68ac4b0bc9975bcb4010.png",
      "src": "https://substackcdn.com/image/fetch/$s_!ie6t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F216100ce-7a2b-4313-b047-25451e29620f_1872x936.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!90RN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97d85f1-23c5-4db8-b087-c738f11cfcb1_1032x400.png",
      "bytes": 40242,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-a4f5fe5b0f1ac7c5cda9.png",
      "src": "https://substackcdn.com/image/fetch/$s_!90RN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97d85f1-23c5-4db8-b087-c738f11cfcb1_1032x400.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Qu_u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F2a242e94-54ba-44e1-9f99-663a8330a67d_1966x800.png",
      "bytes": 116727,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-ac591f1ca70b60eef9b2.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Qu_u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F2a242e94-54ba-44e1-9f99-663a8330a67d_1966x800.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!lPzs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d361dbb-7fba-48cf-8e69-d2f043d6732d_1898x520.png",
      "bytes": 82473,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-dc43bbe17427bd7cf652.png",
      "src": "https://substackcdn.com/image/fetch/$s_!lPzs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d361dbb-7fba-48cf-8e69-d2f043d6732d_1898x520.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!D4E1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e2ebf77-22bd-41e5-a42f-cc815c28f130_2368x636.png",
      "bytes": 80650,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-ded386a4453bc3009ea2.png",
      "src": "https://substackcdn.com/image/fetch/$s_!D4E1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e2ebf77-22bd-41e5-a42f-cc815c28f130_2368x636.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!w89E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf3c018f-b4fe-41fc-aefe-a2c323fa74f2_1524x1350.png",
      "bytes": 292962,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-a9064b273662393f6018.png",
      "src": "https://substackcdn.com/image/fetch/$s_!w89E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf3c018f-b4fe-41fc-aefe-a2c323fa74f2_1524x1350.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!W51N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9cc525df-2702-4ca8-98ff-81d7b70bf56e_1902x1104.png",
      "bytes": 244484,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-7b71d66c96aa3bf30c39.png",
      "src": "https://substackcdn.com/image/fetch/$s_!W51N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9cc525df-2702-4ca8-98ff-81d7b70bf56e_1902x1104.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!a4qS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73777717-0e01-44e8-9f16-c7e93be4896b_2818x818.png",
      "bytes": 93168,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-2f12ec0459dd70d7a9e3.png",
      "src": "https://substackcdn.com/image/fetch/$s_!a4qS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73777717-0e01-44e8-9f16-c7e93be4896b_2818x818.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!SkR-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff49c076a-ce2e-42b4-bffe-8b2a12912851_1860x998.png",
      "bytes": 221529,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-0bcdd010eb54bd8bcf6f.png",
      "src": "https://substackcdn.com/image/fetch/$s_!SkR-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff49c076a-ce2e-42b4-bffe-8b2a12912851_1860x998.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!yhCi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F310c8a15-8acb-4bd6-871a-ebb657386a31_1394x710.png",
      "bytes": 60314,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-9ab09d5e612ad4d86a4f.png",
      "src": "https://substackcdn.com/image/fetch/$s_!yhCi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F310c8a15-8acb-4bd6-871a-ebb657386a31_1394x710.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!siKp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb862c34d-6730-41c5-8071-e79f26fbda5d_1096x772.png",
      "bytes": 77748,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-728f0cc3427691890352.png",
      "src": "https://substackcdn.com/image/fetch/$s_!siKp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb862c34d-6730-41c5-8071-e79f26fbda5d_1096x772.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!NVCM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95f0364e-8bdb-4144-8ff0-cb6c723af58e_1098x708.png",
      "bytes": 53415,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-86c79d0ad380d780b0c3.png",
      "src": "https://substackcdn.com/image/fetch/$s_!NVCM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95f0364e-8bdb-4144-8ff0-cb6c723af58e_1098x708.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!ZaUV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79dd6964-bd10-4aa3-ae43-969c2f83a611_2440x1068.png",
      "bytes": 81021,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-4c409f04e8ca21d270a3.png",
      "src": "https://substackcdn.com/image/fetch/$s_!ZaUV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79dd6964-bd10-4aa3-ae43-969c2f83a611_2440x1068.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!PzpM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93d435bf-c109-44a1-8090-afe85175e184_1470x1200.png",
      "bytes": 198917,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-746aaea321c97f305ea5.png",
      "src": "https://substackcdn.com/image/fetch/$s_!PzpM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93d435bf-c109-44a1-8090-afe85175e184_1470x1200.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!MtzQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a31e405-b7d2-4724-bfe3-ef69912da43d_1904x850.png",
      "bytes": 143096,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-2a900abaa466e164d8ee.png",
      "src": "https://substackcdn.com/image/fetch/$s_!MtzQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a31e405-b7d2-4724-bfe3-ef69912da43d_1904x850.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!gWSa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F248efee3-31f7-4b53-9709-270254e13547_1138x1394.png",
      "bytes": 203359,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-0694697bf08bc379aa49.png",
      "src": "https://substackcdn.com/image/fetch/$s_!gWSa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F248efee3-31f7-4b53-9709-270254e13547_1138x1394.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!L4Aa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a11f01c-02c4-4ede-bf3f-cbda3f5df9a5_1250x840.png",
      "bytes": 193052,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-384d929d82dc2424f3c2.png",
      "src": "https://substackcdn.com/image/fetch/$s_!L4Aa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a11f01c-02c4-4ede-bf3f-cbda3f5df9a5_1250x840.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!GgSm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3fe055b-9ca4-410c-b36d-d6e8047b4b28_1554x924.png",
      "bytes": 62267,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-7c7c25e66e96e3e17682.png",
      "src": "https://substackcdn.com/image/fetch/$s_!GgSm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3fe055b-9ca4-410c-b36d-d6e8047b4b28_1554x924.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!vWyx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81237158-d1dd-4f39-a77b-ea3f70f83a6d_1756x970.png",
      "bytes": 159746,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-e1ac5bc21d6d2a256655.png",
      "src": "https://substackcdn.com/image/fetch/$s_!vWyx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81237158-d1dd-4f39-a77b-ea3f70f83a6d_1756x970.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!kRSV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe851103a-aa40-47cf-b2cc-85f8cee44e06_1274x910.png",
      "bytes": 194006,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-2179d9b2af08c1dc5d74.png",
      "src": "https://substackcdn.com/image/fetch/$s_!kRSV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe851103a-aa40-47cf-b2cc-85f8cee44e06_1274x910.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!NntB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbe15be-8b38-4584-bba9-c9c5e20c5806_1586x1018.png",
      "bytes": 123749,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-1176aae31928a75cc2a2.png",
      "src": "https://substackcdn.com/image/fetch/$s_!NntB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbe15be-8b38-4584-bba9-c9c5e20c5806_1586x1018.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!bO6X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f81e548-b371-4576-a694-a50748f5f148_1586x288.png",
      "bytes": 50686,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-8ed143194630c2c14d6e.png",
      "src": "https://substackcdn.com/image/fetch/$s_!bO6X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f81e548-b371-4576-a694-a50748f5f148_1586x288.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!A46L!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c6aaa13-cbb4-45e9-baa7-f65e6f176538_1014x976.png",
      "bytes": 107617,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-1c4960e5210642d58bf7.png",
      "src": "https://substackcdn.com/image/fetch/$s_!A46L!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c6aaa13-cbb4-45e9-baa7-f65e6f176538_1014x976.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!QxDk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F775634bb-6519-4495-a3b1-83a437b590de_2240x782.png",
      "bytes": 71146,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-c4f85ba4b9be20446cbd.png",
      "src": "https://substackcdn.com/image/fetch/$s_!QxDk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F775634bb-6519-4495-a3b1-83a437b590de_2240x782.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!XmzO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb4408e7-7a56-4ffb-8561-e8261dda1022_2560x1258.png",
      "bytes": 53690,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-e85e3ae327fe32d06ea6.png",
      "src": "https://substackcdn.com/image/fetch/$s_!XmzO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb4408e7-7a56-4ffb-8561-e8261dda1022_2560x1258.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!0EW2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd1a733fb-4558-4ec4-8aa8-f130199a646c_1788x1212.png",
      "bytes": 524970,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-a070c5108f0d6cc9c004.png",
      "src": "https://substackcdn.com/image/fetch/$s_!0EW2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd1a733fb-4558-4ec4-8aa8-f130199a646c_1788x1212.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!B-VP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e7871ad-0e65-4b86-a7bb-3a7b62566e4f_1274x294.png",
      "bytes": 49241,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-ebdde8d8445bd8709dbd.png",
      "src": "https://substackcdn.com/image/fetch/$s_!B-VP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e7871ad-0e65-4b86-a7bb-3a7b62566e4f_1274x294.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!QMa3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafb7f02e-15bf-4445-aec6-f98dd061e36a_1276x726.png",
      "bytes": 172438,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-7ecd661a259f71f5293f.png",
      "src": "https://substackcdn.com/image/fetch/$s_!QMa3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafb7f02e-15bf-4445-aec6-f98dd061e36a_1276x726.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!NPw_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F599a636e-b0b2-4de3-84c8-3edf906bfa82_1616x882.png",
      "bytes": 173790,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-79ad2f7a309d8a6f27f6.png",
      "src": "https://substackcdn.com/image/fetch/$s_!NPw_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F599a636e-b0b2-4de3-84c8-3edf906bfa82_1616x882.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!9A6I!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F896f5300-4bfe-4595-80e7-1063b4225a5b_2400x1650.png",
      "bytes": 227494,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-dd22342b90f1c689bf8b.png",
      "src": "https://substackcdn.com/image/fetch/$s_!9A6I!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F896f5300-4bfe-4595-80e7-1063b4225a5b_2400x1650.png",
      "alt": "Breakdown of the accuracy and raw score of gpt-4o vs. o1 on various competition evals",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!OAWu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9522717b-cade-49a2-a2e4-f60eb1007127_1980x1113.webp",
      "bytes": 59579,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-94100b08a66891f75052.webp",
      "src": "https://substackcdn.com/image/fetch/$s_!OAWu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9522717b-cade-49a2-a2e4-f60eb1007127_1980x1113.webp",
      "alt": "The image shows two scatter plots comparing \"o1 AIME accuracy\" during training and at test time. Both charts have \"pass@1 accuracy\" on the y-axis and compute (log scale) on the x-axis. The dots indicate increasing accuracy with more compute time.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!iCwS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2975bbda-3d86-4cf6-9108-9c20abc9ec59_1286x1300.png",
      "bytes": 134934,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-b5a54ded4b04fc149b24.png",
      "src": "https://substackcdn.com/image/fetch/$s_!iCwS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2975bbda-3d86-4cf6-9108-9c20abc9ec59_1286x1300.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!OfaP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1e98bbb-e1d6-454b-ac9f-c43a38ba0fb5_2516x952.png",
      "bytes": 166170,
      "reused": true,
      "local_path": "attachments/514a31d31a2b/images/body-93871c34508bd12e994f.png",
      "src": "https://substackcdn.com/image/fetch/$s_!OfaP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1e98bbb-e1d6-454b-ac9f-c43a38ba0fb5_2516x952.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=xXCBz_8hM9w",
      "status": "ok",
      "local_path": "attachments/514a31d31a2b/videos/xXCBz_8hM9w.mp4",
      "bytes": 174149694,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "2811.541000"
    },
    {
      "url": "https://www.youtube.com/watch?v=1yvBqasHLZs",
      "status": "ok",
      "local_path": "attachments/514a31d31a2b/videos/1yvBqasHLZs.mp4",
      "bytes": 29330352,
      "valid": true,
      "height": 720,
      "width": 1280,
      "duration": "1476.821000"
    },
    {
      "url": "https://www.youtube.com/watch?v=ugvHCXCOmm4",
      "status": "ok",
      "local_path": "attachments/514a31d31a2b/videos/repair-ugvHCXCOmm4.mp4",
      "bytes": 1433426886,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "18900.041000"
    },
    {
      "url": "https://www.youtube.com/watch?v=pE3KKUKXcTM",
      "status": "ok",
      "local_path": "attachments/514a31d31a2b/videos/repair-pE3KKUKXcTM.mp4",
      "bytes": 1040580922,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "7853.221000"
    },
    {
      "url": "https://www.youtube.com/watch?v=SKBG1sqdyIU",
      "status": "ok",
      "local_path": "attachments/514a31d31a2b/videos/repair-SKBG1sqdyIU.mp4",
      "bytes": 96930675,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "1324.381000"
    }
  ],
  "links": [
    {
      "href": "https://substackcdn.com/image/fetch/$s_!3iKS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98592f1d-1d20-4c88-a681-9d4dac0289d4_2014x1130.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=xXCBz_8hM9w",
      "label": "religious level of belief",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Power_law",
      "label": "Power laws",
      "type": "a"
    },
    {
      "href": "http://felix.physics.sunysb.edu/~allen/540-05/scaling.html",
      "label": "scale-invariant",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!pZed!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b410c6-03f9-4214-8d6a-074b1cbbf6ec_800x300.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!Av0X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82fbae3c-e5d1-4936-8328-057eba9893a3_806x524.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!ybvx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb36a47-1a10-41c1-ad10-dd0a5a7df7d2_800x300.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2001.08361",
      "label": "Scaling Laws for Neural Language Models",
      "type": "a"
    },
    {
      "href": "https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf",
      "label": "original GPT",
      "type": "a"
    },
    {
      "href": "https://github.com/EleutherAI/openwebtext2",
      "label": "WebText2 corpus",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.substack.com/i/136638774/understanding-next-token-prediction",
      "label": "next token prediction",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html",
      "label": "cross-entropy",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!OvRU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F071a53ae-e1d5-4af6-bcd5-4402cc27e924_2144x1244.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.substack.com/i/142044446/constructing-the-models-input",
      "label": "positional or token embeddings",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!0b4b!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe90d52a4-413f-4140-977d-3e0c52497dae_1968x808.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!BX8S!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9603be2-d7fd-4f59-bda2-1e5f76f17f8d_1254x502.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!ELFU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb29aeb01-107f-4b6f-b462-7477f6bd863b_1256x576.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://techcrunch.com/2024/03/27/databricks-spent-10m-on-a-generative-ai-model-that-still-cant-beat-gpt-4/",
      "label": "incredibly expensive",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!aoTo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7bae506a-dddf-440b-be73-87bafe6d7d0f_1884x1032.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2203.03466",
      "label": "various approaches",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.substack.com/p/data-is-the-foundation-of-language",
      "label": "better",
      "type": "a"
    },
    {
      "href": "https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf",
      "label": "GPT-2",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2005.14165",
      "label": "GPT-3",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2303.08774",
      "label": "GPT-4",
      "type": "a"
    },
    {
      "href": "https://www.interconnects.ai/p/frontier-model-post-training",
      "label": "post-training research",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!ie6t!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F216100ce-7a2b-4313-b047-25451e29620f_1872x936.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://paperswithcode.com/dataset/bookcorpus",
      "label": "BooksCorpus",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.substack.com/p/language-model-training-and-inference",
      "label": "self-supervised pretraining",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.substack.com/p/decoder-only-transformers-the-workhorse",
      "label": "decoder-only transformer",
      "type": "a"
    },
    {
      "href": "https://paperswithcode.com/dataset/webtext",
      "label": "WebText",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.substack.com/i/117151147/zero-shot-learning",
      "label": "zero-shot inference",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!Qu_u!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F2a242e94-54ba-44e1-9f99-663a8330a67d_1966x800.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!lPzs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d361dbb-7fba-48cf-8e69-d2f043d6732d_1898x520.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://commoncrawl.org/",
      "label": "CommonCrawl",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!D4E1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e2ebf77-22bd-41e5-a42f-cc815c28f130_2368x636.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.substack.com/i/117151147/few-shot-learning",
      "label": "few-shot learning",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!w89E!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf3c018f-b4fe-41fc-aefe-a2c323fa74f2_1524x1350.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!W51N!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9cc525df-2702-4ca8-98ff-81d7b70bf56e_1902x1104.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!a4qS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73777717-0e01-44e8-9f16-c7e93be4896b_2818x818.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://blogs.nvidia.com/blog/what-are-foundation-models/",
      "label": "foundation models",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/chatgpt/",
      "label": "ChatGPT",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.substack.com/p/understanding-and-using-supervised",
      "label": "supervised finetuning",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.substack.com/p/the-story-of-rlhf-origins-motivations",
      "label": "reinforcement learning from human feedback",
      "type": "a"
    },
    {
      "href": "https://platform.openai.com/docs/api-reference/introduction",
      "label": "their API",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!SkR-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff49c076a-ce2e-42b4-bffe-8b2a12912851_1860x998.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!siKp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb862c34d-6730-41c5-8071-e79f26fbda5d_1096x772.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/human-eval",
      "label": "HumanEval",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!NVCM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95f0364e-8bdb-4144-8ff0-cb6c723af58e_1098x708.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2203.15556",
      "label": "Chinchilla:",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!ZaUV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79dd6964-bd10-4aa3-ae43-969c2f83a611_2440x1068.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!PzpM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93d435bf-c109-44a1-8090-afe85175e184_1470x1200.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!MtzQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a31e405-b7d2-4724-bfe3-ef69912da43d_1904x850.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!gWSa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F248efee3-31f7-4b53-9709-270254e13547_1138x1394.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.reuters.com/technology/artificial-intelligence/openai-rivals-seek-new-path-smarter-ai-current-methods-hit-limitations-2024-11-11",
      "label": "Reuters",
      "type": "a"
    },
    {
      "href": "https://www.theinformation.com/articles/openai-shifts-strategy-as-rate-of-gpt-ai-improvements-slows",
      "label": "The Information",
      "type": "a"
    },
    {
      "href": "https://www.bloomberg.com/news/articles/2024-11-13/openai-google-and-anthropic-are-struggling-to-build-more-advanced-ai",
      "label": "Bloomberg",
      "type": "a"
    },
    {
      "href": "https://techcrunch.com/2024/11/20/ai-scaling-laws-are-showing-diminishing-returns-forcing-ai-labs-to-change-course/",
      "label": "TechCrunch",
      "type": "a"
    },
    {
      "href": "https://time.com/7178328/is-ai-progress-slowing-down/",
      "label": "Time",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=1yvBqasHLZs",
      "label": "test of time award speech",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=ugvHCXCOmm4",
      "label": "Dario Amodei",
      "type": "a"
    },
    {
      "href": "https://x.com/sama/status/1856941766915641580",
      "label": "“there is no wall”",
      "type": "a"
    },
    {
      "href": "https://www.interconnects.ai/p/scaling-realities",
      "label": "Nathan Lambert",
      "type": "a"
    },
    {
      "href": "https://epoch.ai/frontiermath/the-benchmark",
      "label": "advanced mathematical reasoning problems",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!L4Aa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a11f01c-02c4-4ede-bf3f-cbda3f5df9a5_1250x840.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!GgSm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3fe055b-9ca4-410c-b36d-d6e8047b4b28_1554x924.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://docs.google.com/presentation/d/179dpzWSQ9G7EAUlvaJdeE0av9PLuk9Rl33nfhHSJ4xI/edit",
      "label": "source",
      "type": "a"
    },
    {
      "href": "https://www.reuters.com/technology/artificial-intelligence/ai-with-reasoning-power-will-be-less-predictable-ilya-sutskever-says-2024-12-14/",
      "label": "recently made this argument",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!vWyx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81237158-d1dd-4f39-a77b-ea3f70f83a6d_1756x970.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.interconnects.ai/p/llm-synthetic-data",
      "label": "seemingly successful",
      "type": "a"
    },
    {
      "href": "https://www.databricks.com/blog/characterizing-datasets-and-building-better-models-continued-pre-training",
      "label": "continued pretraining",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!kRSV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe851103a-aa40-47cf-b2cc-85f8cee44e06_1274x910.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/mosaicml/llm-foundry",
      "label": "LLM Foundry",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.substack.com/p/conditional-computation-the-birth",
      "label": "mixture-of-experts",
      "type": "a"
    },
    {
      "href": "https://github.com/deepseek-ai/DeepSeek-V3/blob/main/LICENSE-MODEL",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://ai.meta.com/blog/meta-llama-3/",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!NntB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbe15be-8b38-4584-bba9-c9c5e20c5806_1586x1018.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://ai.meta.com/blog/meta-llama-3-1/",
      "label": "largest LLaMA models",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2405.04434",
      "label": "DeepSeek-v2",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.substack.com/i/142423094/switch-transformers-scaling-to-trillion-parameter-models-with-simple-and-efficient-sparsity",
      "label": "load balancing",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2404.19737",
      "label": "multi-token prediction",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/learning-to-reason-with-llms/",
      "label": "OpenAI’s o1",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=pE3KKUKXcTM",
      "label": "potentially spanning multiple data centers",
      "type": "a"
    },
    {
      "href": "https://scholar.google.com/citations?user=x4aaIwcAAAAJ&hl=en",
      "label": "Ege Erdil",
      "type": "a"
    },
    {
      "href": "https://epoch.ai/",
      "label": "Epoch AI",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!A46L!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c6aaa13-cbb4-45e9-baa7-f65e6f176538_1014x976.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!QxDk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F775634bb-6519-4495-a3b1-83a437b590de_2240x782.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/summarizing-books/",
      "label": "recursive / hierarchical decomposition",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!XmzO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb4408e7-7a56-4ffb-8561-e8261dda1022_2560x1258.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://menlovc.com/2024-the-state-of-generative-ai-in-the-enterprise/",
      "label": "source",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!0EW2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd1a733fb-4558-4ec4-8aa8-f130199a646c_1788x1212.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.interconnects.ai/p/the-ai-agent-spectrum",
      "label": "overloaded",
      "type": "a"
    },
    {
      "href": "https://www.interconnects.ai/p/how-scaling-changes-model-behavior",
      "label": "Recent research",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!QMa3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafb7f02e-15bf-4445-aec6-f98dd061e36a_1276x726.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.substack.com/p/chain-of-thought-prompting-for-llms",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2205.11916",
      "label": "step-by-step",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!NPw_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F599a636e-b0b2-4de3-84c8-3edf906bfa82_1616x882.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://x.com/__nmca__/status/1870170098989674833",
      "label": "public comments",
      "type": "a"
    },
    {
      "href": "https://www.interconnects.ai/p/openais-o1-using-search-was-a-psyop",
      "label": "recent rhetoric",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!9A6I!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F896f5300-4bfe-4595-80e7-1063b4225a5b_2400x1650.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://codeforces.com/",
      "label": "Codeforces",
      "type": "a"
    },
    {
      "href": "https://artofproblemsolving.com/wiki/index.php/American_Invitational_Mathematics_Examination?srsltid=AfmBOoplGn5Sl-tAVYpjGeZiZChoPCs-y6Ofqk90mzlvesrqhkA5kL6s",
      "label": "AIME",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2311.12022",
      "label": "GPQA",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!OAWu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9522717b-cade-49a2-a2e4-f60eb1007127_1980x1113.webp",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=SKBG1sqdyIU&t=2s",
      "label": "previewed",
      "type": "a"
    },
    {
      "href": "https://www.interconnects.ai/p/openais-o3-the-2024-finale-of-ai",
      "label": "“scaled up” version of o1",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!iCwS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2975bbda-3d86-4cf6-9108-9c20abc9ec59_1286x1300.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arcprize.org/blog/oai-o3-pub-breakthrough",
      "label": "source",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/introducing-swe-bench-verified/",
      "label": "SWE-Bench Verified",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Elo_rating_system",
      "label": "Elo score",
      "type": "a"
    },
    {
      "href": "https://epoch.ai/frontiermath",
      "label": "FrontierMath benchmark",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Terence_Tao",
      "label": "Terence Tao",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!OfaP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1e98bbb-e1d6-454b-ac9f-c43a38ba0fb5_2516x952.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://cameronrwolfe.me/",
      "label": "Cameron R. Wolfe",
      "type": "a"
    },
    {
      "href": "https://research.netflix.com/research-area/nlp-and-conversations",
      "label": "Netflix",
      "type": "a"
    },
    {
      "href": "https://twitter.com/cwolferesearch",
      "label": "X",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/cameron-r-wolfe-ph-d-04744a238/",
      "label": "LinkedIn",
      "type": "a"
    },
    {
      "href": "https://www.reuters.com/technology/artificial-intelligence/openai-rivals-seek-new-path-smarter-ai-current-methods-hit-limitations-2024-11-11/",
      "label": "Reuters",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1409.3215",
      "label": "this paper",
      "type": "a"
    },
    {
      "href": "https://www.anthropic.com/news/claude-3-5-sonnet",
      "label": "Claude 3.5 Opus",
      "type": "a"
    },
    {
      "href": "https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/",
      "label": "flash variant of Gemini-2",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/hello-gpt-4o/",
      "label": "GPT-4o",
      "type": "a"
    },
    {
      "href": "https://openai.com/o1/",
      "label": "o1",
      "type": "a"
    },
    {
      "href": "https://www.datacenterfrontier.com/machine-learning/article/55244139/the-colossus-ai-supercomputer-elon-musks-drive-toward-data-center-ai-technology-domination",
      "label": "new datacenter",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=ugvHCXCOmm4&t=1103s",
      "label": "voiced a desire",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-18T17:01:30Z",
  "source_url": "https://cameronrwolfe.substack.com/p/llm-scaling-laws",
  "delivery_status": "ok",
  "discovery_status": "ok",
  "missing_reason": "",
  "repair_error": "",
  "actual_video_files": [
    {
      "local_path": "attachments/514a31d31a2b/videos/1yvBqasHLZs.mp4",
      "bytes": 29330352,
      "valid": true,
      "height": 720,
      "width": 1280,
      "duration": "1476.821000",
      "status": "ok"
    },
    {
      "local_path": "attachments/514a31d31a2b/videos/repair-SKBG1sqdyIU.mp4",
      "bytes": 96930675,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "1324.381000",
      "status": "ok"
    },
    {
      "local_path": "attachments/514a31d31a2b/videos/repair-pE3KKUKXcTM.mp4",
      "bytes": 1040580922,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "7853.221000",
      "status": "ok"
    },
    {
      "local_path": "attachments/514a31d31a2b/videos/repair-ugvHCXCOmm4.mp4",
      "bytes": 1433426886,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "18900.041000",
      "status": "ok"
    },
    {
      "local_path": "attachments/514a31d31a2b/videos/xXCBz_8hM9w.mp4",
      "bytes": 174149694,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "2811.541000",
      "status": "ok"
    }
  ],
  "expected_saved_body_images": 48,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "[![Deep \\(Learning\\) Focus](https://substackcdn.com/image/fetch/$s_!87xa!,w_40,h_40,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fab9b43fb-52d5-40da-995d-5b7cd3f91064_896x896.png)](https://cameronrwolfe.substack.com/)\n# [Deep (Learning) Focus](https://cameronrwolfe.substack.com/)\nSubscribeSign in\n![User's avatar](https://substackcdn.com/image/fetch/$s_!VC2M!,w_64,h_64,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F69aba7df-b571-4609-aa47-fc2d031c11b8_1242x1595.jpeg)\nDiscover more from Deep (Learning) Focus\nI contextualize and explain important topics in AI research.\nOver 76,000 subscribers\nSubscribe\nBy subscribing, you agree Substack's [Terms of Use](https://substack.com/tos), and acknowledge its [Information Collection Notice](https://substack.com/ccpa#personal-data-collected) and [Privacy Policy](https://substack.com/privacy).\nAlready have an account? Sign in\n# Scaling Laws for LLMs: From GPT-3 to o3\n### Understanding the current state of LLM scaling and the future of AI research...\n[![Cameron R. Wolfe, Ph.D.'s avatar](https://substackcdn.com/image/fetch/$s_!VC2M!,w_36,h_36,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F69aba7df-b571-4609-aa47-fc2d031c11b8_1242x1595.jpeg)](https://substack.com/@cwolferesearch)\n[Cameron R. Wolfe, Ph.D.](https://substack.com/@cwolferesearch)\nJan 06, 2025\n155\n10\n17\nShare\n[![](https://substackcdn.com/image/fetch/$s_!3iKS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98592f1d-1d20-4c88-a681-9d4dac0289d4_2014x1130.png)](https://substackcdn.com/image/fetch/$s_!3iKS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98592f1d-1d20-4c88-a681-9d4dac0289d4_2014x1130.png)(from [1, 7, 10, 21])\nA majority of recent advancements in AI research— _and large language models (LLMs) in particular_ —have been driven by scale. If we train larger models over more data, we get better results. This relationship can be defined more rigorously via a scaling law, which is just an equation that describes how an LLM’s test loss will decrease as we increase some quantity of interest (e.g., training compute). Scaling laws help us to predict the results of larger and more expensive training runs, giving us the necessary confidence to continue investing in scale.\n> _“If you have a large dataset and you train a very big neural network, then success is guaranteed!”_ - Ilya Sutskever\nFor years, scaling laws have been a predictable North Star for AI research. In fact, the success of early frontier labs like OpenAI has even been credited to their[ religious level of belief](https://www.youtube.com/watch?v=xXCBz_8hM9w) in scaling laws. However, the continuation of scaling has recently been called into question by reports[1](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-1) claiming that top research labs are struggling to create the next generation of better LLMs. These claims might lead us to wonder: _Will scaling hit a wall and, if so, are there other paths forward?_\nThis overview will answer these questions from the ground up, beginning with an in-depth explanation of LLM scaling laws and the surrounding research. The idea of a scaling law is simple, but there are a variety of public misconceptions around scaling— _the science behind this research is actually very specific_. Using this detailed understanding of scaling, we will then discuss recent trends in LLM research and contributing factors to the “plateau” of scaling laws. Finally, we will use this information to more clearly illustrate the future of AI research, focusing on a few key ideas— _including scaling_ —that could continue to drive progress.\n## Fundamental Scaling Concepts for LLMs\nTo understand the state of scaling for LLMs, we first need to build a general understanding of scaling laws. We will build this understanding from the ground up, starting with the concept of a power law. Then, we will explore how power laws have been applied in LLM research to derive the scaling laws we use today.\n#### What is a power law?\n[Power laws](https://en.wikipedia.org/wiki/Power_law) are the fundamental concept that underlie LLM scaling. Put simply, power laws just describe a relationship between two quantities. For LLMs, the first of these quantities is the LLM’s test loss— _or some other related performance metric (e.g., downstream task accuracy [7])_ —and the other is some setting that we are trying to scale, such as the number of model parameters. For example, we may see a statement like the following when studying the scaling properties of an LLM.\n> _“With enough training data, scaling of validation loss should be approximately a smooth power law as a function of model size.”_ - from [4]\nSuch a statement tells us that a measurable relationship exists between the model’s test loss and the total number of model parameters. A change to one of these quantities will produce a relative, [scale-invariant](http://felix.physics.sunysb.edu/~allen/540-05/scaling.html#:~:text=scale%20invariance%20of%20power%20law%20functions&text=The%20function%20y%3Dxp,by%20a%20scale%20factor%20a.) change in the other. In other words, we know from this relationship that increasing the total number of model parameters— _assuming other conditions (e.g., having sufficient training data) are met_ —will result in a decrease of the test loss by predictable factor.\n**Power law formulation.** A basic power law is expressed via the equation below.\n[![](https://substackcdn.com/image/fetch/$s_!VKCD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4e231b5-3002-4bec-a733-512e3f6730a5_240x68.png)](https://substackcdn.com/image/fetch/$s_!VKCD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4e231b5-3002-4bec-a733-512e3f6730a5_240x68.png)\nThe two quantities being studied here are `x` and `y`, while `a` and `p` are constants that describe the relationship between these quantities. If we plot this power law function[2](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-2), we get the figure shown below. We provide plots in both normal and log scale because most papers that study LLM scaling use a log scale.\n[![](https://substackcdn.com/image/fetch/$s_!pZed!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b410c6-03f9-4214-8d6a-074b1cbbf6ec_800x300.png)](https://substackcdn.com/image/fetch/$s_!pZed!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b410c6-03f9-4214-8d6a-074b1cbbf6ec_800x300.png)Plot of a basic power law between `x` and `y`\nHowever, the plots provided for LLM scaling do not look like the plot shown above— _they are usually flipped upside down_ ; see below for an example.\n[![](https://substackcdn.com/image/fetch/$s_!Av0X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82fbae3c-e5d1-4936-8328-057eba9893a3_806x524.png)](https://substackcdn.com/image/fetch/$s_!Av0X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82fbae3c-e5d1-4936-8328-057eba9893a3_806x524.png)(from [1])\nThis is just an _inverse_ power law, which can be formulated as shown below.\n[![](https://substackcdn.com/image/fetch/$s_!EmoD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19681045-c6e7-41dc-aee7-8fb87ec02b5f_376x178.png)](https://substackcdn.com/image/fetch/$s_!EmoD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19681045-c6e7-41dc-aee7-8fb87ec02b5f_376x178.png)\nThe equation for an inverse power law is nearly identically to that of a standard power law, but we use a negative exponent for `p`. Making the exponent of the power law negative flips the plots upside down; see below for an example.\n[![](https://substackcdn.com/image/fetch/$s_!ybvx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb36a47-1a10-41c1-ad10-dd0a5a7df7d2_800x300.png)](https://substackcdn.com/image/fetch/$s_!ybvx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb36a47-1a10-41c1-ad10-dd0a5a7df7d2_800x300.png)Plot of an inverse power law between `x` and `y`\nThis inverse power law, when plotted using a log scale, yields the signature linear relationship that is characteristic of most LLM scaling laws. Nearly every paper covered in this overview will produce such a plot to study how scaling up a variety of different factors (e.g., size, compute, data, etc.) impacts an LLM’s performance. Now, let’s take a more practical look at power laws by learning about one of the first papers to study them in the context of LLM scaling [1].\n#### **[Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361) [1]**\nIn the early days of language models, we did not yet understand the impact of scale on performance. Language models were a promising area of research, but the current generation of models (e.g., the [original GPT](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)) at that time had limited capabilities. We had yet to discover the power of larger models, and the path towards creating better language models was not immediately clear. _Is the shape of the model (i.e., the number and size of layers) important? Does making a model larger help it to perform better? How much data is needed to train these larger models?_\n> _“The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.”_ - from [1]\nIn [1], authors aim to answer these questions by analyzing the impact of several factors— _such as model size, model shape, dataset size, training compute, and batch size_ —on model performance. From this analysis, we learn that LLM performance improves smoothly as we increase:\n  1. The number of model parameters.\n  2. The size of the dataset.\n  3. The amount of compute used for training.\n\n\nMore specifically, _a power law relationship is observed between each of these factors and the LLM’s test loss when performance is not bottlenecked by the other two factors._\n**Experimental setup.** To fit their power laws, authors pretrain LLMs with sizes up to 1.5B parameters over subsets of the [WebText2 corpus](https://github.com/EleutherAI/openwebtext2), containing anywhere from 22M to 23B tokens. All models are trained using a fixed context length of 1,024 tokens and a standard [next token prediction](https://cameronrwolfe.substack.com/i/136638774/understanding-next-token-prediction) ([cross-entropy](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)) loss. The same loss is measured over a hold-out test set and used as our primary performance metric. _This setup matches the standard pretraining setup for most LLMs_. \n[![](https://substackcdn.com/image/fetch/$s_!OvRU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F071a53ae-e1d5-4af6-bcd5-4402cc27e924_2144x1244.png)](https://substackcdn.com/image/fetch/$s_!OvRU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F071a53ae-e1d5-4af6-bcd5-4402cc27e924_2144x1244.png)(from [1])\n**Power laws for LLM scaling.** The performance of LLMs trained in [1]— _in terms of their test loss on WebText2_ —is shown to steadily improve with more parameters, data, and compute[3](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-3). These trends span eight orders of magnitude in compute, six orders of magnitude in model size, and two orders of magnitude in dataset size. The exact power law relationship and the equations that are fit to each of them are provided in the figure above. Each of the equations here are very similar to the inverse power law equation that we saw before. However, we set `a = 1` and add an additional multiplicative constant inside of the parenthesis[4](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-4).\nAuthors in [1] note one small detail that is necessary to properly fit these power laws. We do not include [positional or token embeddings](https://cameronrwolfe.substack.com/i/142044446/constructing-the-models-input) when counting the total number of model parameters, which yields cleaner scaling trends; see below.\n[![](https://substackcdn.com/image/fetch/$s_!0b4b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe90d52a4-413f-4140-977d-3e0c52497dae_1968x808.png)](https://substackcdn.com/image/fetch/$s_!0b4b!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe90d52a4-413f-4140-977d-3e0c52497dae_1968x808.png)(from [1])\nThese power laws are only applicable if training is not bottlenecked by the other factors. So, all three of these components— _model size, data, and compute_ —should be scaled up simultaneously for optimal performance. If we scale up any of these components in isolation, we will reach a point of diminishing returns.\n**What do power laws tell us?** Although the power law plots provided in [1] look quite promising, we should notice that these plots are generated using a log scale. If we generate normal plots (i.e., without log scale), we get the figures below, where we see that the shape of the power law resembles an exponential decay. \n[![](https://substackcdn.com/image/fetch/$s_!L-l2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabf07f96-f9d9-467f-a459-fb24788f4e33_2540x630.png)](https://substackcdn.com/image/fetch/$s_!L-l2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabf07f96-f9d9-467f-a459-fb24788f4e33_2540x630.png)Power law plots without log scale\nSuch a finding may seem counterintuitive given a lot of the online rhetoric around scaling and AGI. In many cases, it seems like the intuition we are being fed is that the quality LLMs improves exponentially with logarithmic increases in compute, but this is not the case whatsoever. In reality, _increasing the quality of an LLM becomes exponentially more difficult with scale_.\n[![](https://substackcdn.com/image/fetch/$s_!BX8S!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9603be2-d7fd-4f59-bda2-1e5f76f17f8d_1254x502.png)](https://substackcdn.com/image/fetch/$s_!BX8S!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9603be2-d7fd-4f59-bda2-1e5f76f17f8d_1254x502.png)(from [1])\n**Other useful findings.** Beyond the power laws observed in [1], we also see that the other factors that are considered, such as model shape or architecture settings, have minimal influence on model performance; see above. Scale is by far the largest contributing factor for creating better LLMs— _more data, compute, and model parameters yields a smooth improvement in the LLM’s performance_.\n> _“Larger models are significantly more sampleefficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.”_ - from [1]\nInterestingly, the empirical analysis in [1] indicates that larger LLMs tend to be more sample efficient, meaning that they reach the same level of test loss with less data relative to smaller models. For this reason, _pretraining an LLM to convergence is (arguably) sub-optimal_. Instead, we could train a much larger model on less data, stopping the training process far before convergence. Such an approach is optimal in terms of the amount of training compute used but it does not account for inference costs. Practically speaking, we usually train smaller models over more data because smaller models are cheaper to host.\nAuthors also extensively analyze the relationship between model size and the amount of data used for pretraining, finding that the size of the dataset does not need to be increased as quickly as the model size. _An ~8X increase in model size requires a ~5X increase in the amount of training data to avoid overfitting_.\n[![](https://substackcdn.com/image/fetch/$s_!ELFU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb29aeb01-107f-4b6f-b462-7477f6bd863b_1256x576.png)](https://substackcdn.com/image/fetch/$s_!ELFU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb29aeb01-107f-4b6f-b462-7477f6bd863b_1256x576.png)(from [1])\nThe scaling laws discovered in [1] are also replicated on several other datasets, where we see that the same scaling laws hold after adding a fixed offset to the test loss (i.e., to account for the fact that the dataset is different); see above. These results make a compelling case for LLM scaling. We achieve very clear and measurable benefits from training larger models longer and on more data, which created an appetite for pretraining LLMs at a much larger scale.\n> _“These results show that language modeling performance improves smoothly and predictably as we appropriately scale up model size, data, and compute. We expect that larger language models will perform better and be more sample efficient than current models.”_ - from [1]\n#### Practical Usage of Scaling Laws\nThe fact that large-scale pretraining is so beneficial presents us with a slight dilemma. The best results are achieved by training massive models over copious amounts of data. However, these training runs are [incredibly expensive](https://techcrunch.com/2024/03/27/databricks-spent-10m-on-a-generative-ai-model-that-still-cant-beat-gpt-4/), _which means they also incur a lot of risk_. What if we spend $10 million training a model that fails to meet our expectations? Given the expense of pretraining, we cannot perform any model-specific tuning, and we must be sure that the model we train will perform well. We need to develop a strategy for tuning these models and forecasting their performance without spending too much money. \n[![](https://substackcdn.com/image/fetch/$s_!aoTo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7bae506a-dddf-440b-be73-87bafe6d7d0f_1884x1032.png)](https://substackcdn.com/image/fetch/$s_!aoTo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7bae506a-dddf-440b-be73-87bafe6d7d0f_1884x1032.png)(from [11])\nThis is where scaling laws come in. So far, we have seen some empirical analysis that was conducted to prove that scaling laws exist, _but these scaling laws also have a very practical use case within AI research_. In particular, we can:\n  * Train a bunch of smaller models using various training settings.\n  * Fit scaling laws based on the performance of smaller models.\n  * Use the scaling law to extrapolate the performance of a much larger model.\n\n\nOf course, this approach has limitations. Predicting the performance of larger models from smaller models is difficult and can be inaccurate. Models may behave differently depending on scale. However, [various approaches](https://arxiv.org/abs/2203.03466) have been proposed to make this more feasible, and scaling laws are now commonly used for this purpose. The ability to forecast the performance of larger models using scaling laws gives us more confidence (and peace of mind) as researchers. Plus, scaling laws provide a simple way to justify investment into AI research.\n## Scaling and The Age of Pretraining\n> _“This is what has been the driver of all progress we see today—extraordinarily large neural networks trained on huge datasets.”_ - Ilya Sutskever\nThe discovery of scaling laws catalyzed much of the recent progress in LLM research. To get better results, we just train increasingly large models on bigger (and [better](https://cameronrwolfe.substack.com/p/data-is-the-foundation-of-language)!) datasets. This strategy was followed to create several models within the GPT lineage, as well as most of the notable models from groups other than OpenAI. Here, we will take a deeper look at this progression of scaling research— _recently described by Ilya Sutskever as the “age of pretraining”_[5](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-5).\n#### **The GPT Lineage:[GPT](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf) [2], [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) [3], [GPT-3](https://arxiv.org/abs/2005.14165) [4], and [GPT-4](https://arxiv.org/abs/2303.08774) [5]**\nThe most widely known and visible application of LLM scaling laws was in the creation of OpenAI’s GPT lineage of models. We will primarily focus open earlier models in this lineage— _up to GPT-3_ —because:\n  1. The details of these models were shared more openly.\n  2. Later models heavily benefitted from advancements in [post-training research](https://www.interconnects.ai/p/frontier-model-post-training) in addition to scaling up the pretraining process.\n\n\nWe will also cover some of the known scaling results from models like GPT-4.\n[![](https://substackcdn.com/image/fetch/$s_!ie6t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F216100ce-7a2b-4313-b047-25451e29620f_1872x936.png)](https://substackcdn.com/image/fetch/$s_!ie6t!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F216100ce-7a2b-4313-b047-25451e29620f_1872x936.png)(from [2])\n**The original GPT model** [2] was actually quite small— _12 layers and 117M parameters in total_. This model is first pretrained over [BooksCorpus](https://paperswithcode.com/dataset/bookcorpus), a dataset that contains the raw text of ~7,000 books. Then, we finetune the model to solve a variety of different downstream tasks by using a supervised training objective and creating a separate classification head for each task; see above. This paper was one of the first to perform large-scale, [self-supervised pretraining](https://cameronrwolfe.substack.com/p/language-model-training-and-inference) of a [decoder-only transformer](https://cameronrwolfe.substack.com/p/decoder-only-transformers-the-workhorse), which led to a few interesting findings:\n  * Self-supervised pretraining on flat text is incredibly effective.\n  * Using long, contiguous spans of text for pretraining is important.\n  * When pretrained in this manner, a single model can be finetuned to solve a wide variety of different tasks with state-of-the-art accuracy[6](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-6).\n\n\nOverall, GPT was not an especially noteworthy model, but it laid some important foundations (i.e., decoder-only transformer and self-supervised pretraining) for later work that explored similar models at a much larger scale.\n[![](https://substackcdn.com/image/fetch/$s_!90RN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97d85f1-23c5-4db8-b087-c738f11cfcb1_1032x400.png)](https://substackcdn.com/image/fetch/$s_!90RN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97d85f1-23c5-4db8-b087-c738f11cfcb1_1032x400.png)(from [3])\n**GPT-2** [3] was proposed shortly after GPT and includes a collection of several models with sizes up to 1.5B parameters; see above. These models share the same architecture as the GPT model and are pretrained using the same self-supervised language modeling objective. However, GPT-2 makes two big changes to the pretraining process compared to GPT:\n  * The models are pretrained on [WebText](https://paperswithcode.com/dataset/webtext), which is _i)_ much larger than BooksCorpus and _ii)_ created by scraping data from the internet. \n  * The models are not finetuned on downstream tasks. Instead, we solve tasks by performing [zero-shot inference](https://cameronrwolfe.substack.com/i/117151147/zero-shot-learning)[7](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-7) with the pretrained model.\n\n\nGPT-2 models fall short of state-of-the-art performance on most benchmarks[8](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-8), but their performance consistently improves with the size of the model— _scaling up the number of model parameters yields a clear benefit_ ; see below.\n[![](https://substackcdn.com/image/fetch/$s_!Qu_u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F2a242e94-54ba-44e1-9f99-663a8330a67d_1966x800.png)](https://substackcdn.com/image/fetch/$s_!Qu_u!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F2a242e94-54ba-44e1-9f99-663a8330a67d_1966x800.png)(from [3])\nAuthors in [3] also reveal that GPT-2 models— _despite their impressive results_ —still seem to underfit the WebText corpus. From this finding, we can infer that continued scaling of LLM pretraining— _in terms of both model and data size_ —should be beneficial. Although the GPT-2 models were not particularly powerful, the analysis presented by these models provided the confidence we needed to continue scaling and eventually reach an inflection point in AI research.\n> _“A language model with sufficient capacity will begin to learn to infer and perform the tasks demonstrated in natural language sequences in order to better predict them, regardless of their method of procurement.”_ - from [3]\n**GPT-3** [4] was a watershed moment in AI research that definitively confirmed the benefit of large-scale pretraining for LLMs. The model has over 175B parameters, making it over 100x larger than the largest GPT-2 model; see below.\n[![](https://substackcdn.com/image/fetch/$s_!lPzs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d361dbb-7fba-48cf-8e69-d2f043d6732d_1898x520.png)](https://substackcdn.com/image/fetch/$s_!lPzs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d361dbb-7fba-48cf-8e69-d2f043d6732d_1898x520.png)(from [4])\nAgain, GPT-3 uses a decoder-only model architecture that is quite similar to that of prior models, but we pretrain the model over a much larger dataset based upon [CommonCrawl](https://commoncrawl.org/). This dataset is ~10x bigger than the prior WebText dataset, and authors in [4] combine the larger pretraining dataset with several other sources of pretraining data, creating a mixture of different corpora; see below.\n[![](https://substackcdn.com/image/fetch/$s_!D4E1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e2ebf77-22bd-41e5-a42f-cc815c28f130_2368x636.png)](https://substackcdn.com/image/fetch/$s_!D4E1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e2ebf77-22bd-41e5-a42f-cc815c28f130_2368x636.png)(from [4])\nGPT-3 is primarily evaluated in [4] by using a [few-shot learning](https://cameronrwolfe.substack.com/i/117151147/few-shot-learning) approach. The differences between few-shot prompting (used by GPT-3), zero-shot prompting (used by GPT-2), and finetuning (used by GPT) are illustrated below.\n[![](https://substackcdn.com/image/fetch/$s_!w89E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf3c018f-b4fe-41fc-aefe-a2c323fa74f2_1524x1350.png)](https://substackcdn.com/image/fetch/$s_!w89E!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf3c018f-b4fe-41fc-aefe-a2c323fa74f2_1524x1350.png)(from [4])\nFew-shot learning is a new paradigm in which the LLM learns how to perform a task based upon examples that are placed within its context window. Authors in [4] refer to this concept as “in-context learning”. In this case, the LLM does not actually “learn”— _the model’s weights are not updated at all_. Rather, the examples in the model’s input are used as context for generating a more accurate output. We see in [4] that GPT-3 is a highly capable few-shot learner, seeming to indicate that in-context learning is an emergent[9](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-9) ability of larger models; see below.\n[![](https://substackcdn.com/image/fetch/$s_!W51N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9cc525df-2702-4ca8-98ff-81d7b70bf56e_1902x1104.png)](https://substackcdn.com/image/fetch/$s_!W51N!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9cc525df-2702-4ca8-98ff-81d7b70bf56e_1902x1104.png)(from [4])\nWhen GPT-3 is evaluated on various language understanding tasks, we see that using a larger model significantly benefits few-shot learning performance, as shown in the figures below. Larger models make better— _and more efficient_ —use of the information in their context windows relative to smaller models. GPT-3 is able to surpass state-of-the-art performance on several tasks via few-shot learning, and the model’s performance improves smoothly with size.\n[![](https://substackcdn.com/image/fetch/$s_!a4qS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73777717-0e01-44e8-9f16-c7e93be4896b_2818x818.png)](https://substackcdn.com/image/fetch/$s_!a4qS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73777717-0e01-44e8-9f16-c7e93be4896b_2818x818.png)(from [4])\nThe fact that a single model was able to perform this well across so many tasks was incredibly impressive at the time. Solving each of these tasks did not require any finetuning or changes to the underlying model— _we just had to tweak the model’s prompt_. GPT-3 was one of the first true [foundation models](https://blogs.nvidia.com/blog/what-are-foundation-models/) to be released. This model ushered the next era of AI research and introduced a completely new and intuitive paradigm for interacting with LLMs (i.e., prompting).\n**Beyond GPT-3.** The impressive performance of GPT-3 created an explosion of interest in LLM research, primarily focused upon large-scale pretraining. The next few models released by OpenAI— _InstructGPT [8],[ChatGPT](https://openai.com/index/chatgpt/) and GPT-4 [5]_—used a combination of large-scale pretraining and new post-training techniques (i.e., [supervised finetuning](https://cameronrwolfe.substack.com/p/understanding-and-using-supervised) and [reinforcement learning from human feedback](https://cameronrwolfe.substack.com/p/the-story-of-rlhf-origins-motivations)) to drastically improve LLM quality. These models were so impressive that they even caused the amount of public interest in AI research to skyrocket. \n> _“GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior.”_ - from [5] \nAt this time, OpenAI began to publish fewer details on their research. Instead, new models were just released via [their API](https://platform.openai.com/docs/api-reference/introduction), which kept the public from learning how these models were created. Luckily, some useful information can be gleaned from the material that OpenAI did publish. For example, InstructGPT [8]— _the predecessor to ChatGPT_ —has an associated paper that documents the model’s post training strategy in detail; see below. Given that this paper also states that GPT-3 is the base model for InstructGPT, we can reasonably infer that this model’s gains in performance are mostly unrelated to scaling up the pretraining process.\n[![](https://substackcdn.com/image/fetch/$s_!SkR-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff49c076a-ce2e-42b4-bffe-8b2a12912851_1860x998.png)](https://substackcdn.com/image/fetch/$s_!SkR-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff49c076a-ce2e-42b4-bffe-8b2a12912851_1860x998.png)(from [8])\nCompared to ChatGPT, GPT-4 had a noticeable improvement in capabilities. However, researchers chose to share very few of GPT-4’s technical details. The technical report for GPT-4 [5] simply tells us that:\n  1. GPT-4 is transformer-based.\n  2. The model is pretrained using next token prediction.\n  3. Both public and licensed 3rd party data is used.\n  4. The model is finetuned with reinforcement learning from human feedback.\n\n\nNonetheless, the importance of scaling is very evident within this technical report. Authors note that a key challenge within this work was developing a scalable training architecture that behaves predictably across different scales, allowing the results of smaller runs to be extrapolated to provide confidence in the larger-scale (and significantly more expensive!) training exercises. \n> _“The final loss of properly-trained large language models is … approximated by power laws in the amount of compute used to train the model.”_ - from [5]\nLarge-scale pretraining is incredibly expensive, so we usually only get one chance at getting it right— _there is no room for model-specific tuning_. Scaling laws play a key role in this process. We can train models using 1,000-10,000x less compute and use the results of these training runs to fit power laws. Then, these power laws can be used to predict the performance of much larger models. In particular, we see in [8] that the performance of GPT-4 is predicted using a power law that measures the relationship between compute and test loss; see below.\n[![](https://substackcdn.com/image/fetch/$s_!yhCi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F310c8a15-8acb-4bd6-871a-ebb657386a31_1394x710.png)](https://substackcdn.com/image/fetch/$s_!yhCi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F310c8a15-8acb-4bd6-871a-ebb657386a31_1394x710.png)Scaling law formulation for training GPT-4 (from [5])\nThis expression looks nearly identical to what we have seen before, but it has an added irreducible loss term to account for the fact that the LLM’s test loss might never reach zero. Once fitted, the scaling law was used to predict the final performance of GPT-4 with very high accuracy; see below for a depiction. Here, we should note that the plot is not generated using a log scale, and we see that the improvement in loss clearly begins to decay with increasing compute!\n[![](https://substackcdn.com/image/fetch/$s_!siKp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb862c34d-6730-41c5-8071-e79f26fbda5d_1096x772.png)](https://substackcdn.com/image/fetch/$s_!siKp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb862c34d-6730-41c5-8071-e79f26fbda5d_1096x772.png)(from [5])\nAuthors in [5] also note that the test loss is not an easily interpretable metric and try to predict a variety of other performance metrics. For example, a scaling law is fit to predict the LLM’s pass rate on the [HumanEval](https://github.com/openai/human-eval) coding benchmark. First, the problems in HumanEval are split into buckets based on their difficulty. Then, a scaling law is fit to predict the LLM’s pass rate. We see in [5] that the pass rate of GPT-4 can be accurately predicted on HumanEval using this approach based on experiments that require 1,000x less compute; see below.\n[![](https://substackcdn.com/image/fetch/$s_!NVCM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95f0364e-8bdb-4144-8ff0-cb6c723af58e_1098x708.png)](https://substackcdn.com/image/fetch/$s_!NVCM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95f0364e-8bdb-4144-8ff0-cb6c723af58e_1098x708.png)(from [5])\nAs we can see, scaling up the pretraining process is valuable. However, large-scale pretraining is also very expensive. _Scaling laws make this process more predictable_ , allowing us to avoid needless or excessive compute costs. \n####  [Chinchilla: ](https://arxiv.org/abs/2203.15556)**[Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556) [5]**\n[![](https://substackcdn.com/image/fetch/$s_!ZaUV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79dd6964-bd10-4aa3-ae43-969c2f83a611_2440x1068.png)](https://substackcdn.com/image/fetch/$s_!ZaUV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79dd6964-bd10-4aa3-ae43-969c2f83a611_2440x1068.png)(from [9])\nIn [1], authors recommend increasing model size faster than the size of the dataset when scaling up LLM pretraining. However, most of the pretraining research after GPT-3 indicated that we should do the opposite. We trained models that were significantly larger than GPT-3— _such as the 530B parameter MT-NLG [9] model_ —but the size of the dataset used to train these models was similar to that of GPT-3; see above. These models did not improve upon the performance of GPT-3, whereas models that used a combination of more parameters with more data (e.g., Gopher [10]) performed much better; see below.\n[![](https://substackcdn.com/image/fetch/$s_!PzpM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93d435bf-c109-44a1-8090-afe85175e184_1470x1200.png)](https://substackcdn.com/image/fetch/$s_!PzpM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93d435bf-c109-44a1-8090-afe85175e184_1470x1200.png)(from [10])\n**Compute-optimal scaling laws.** Inspired by these observations, authors in [6] completely reconsidered the best practices for scaling laws that were originally proposed in [1]. The analysis of scaling laws in [6] was conducted with much larger models, yielding results that are slightly different from before. More specifically, LLMs with sizes ranging from 70M to 17B parameters are trained over datasets with sizes exceeding one trillion tokens; see below.\n[![](https://substackcdn.com/image/fetch/$s_!MtzQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a31e405-b7d2-4724-bfe3-ef69912da43d_1904x850.png)](https://substackcdn.com/image/fetch/$s_!MtzQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a31e405-b7d2-4724-bfe3-ef69912da43d_1904x850.png)(from [10])\nBy training LLMs with many different combinations of model and data size, we can discover a power law that predicts an LLM’s test loss as a function of these factors. From these power laws, we can determine which training settings work best for a given compute budget. Authors in [6] argue that compute-optimal training[10](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-10) should scale model and data size proportionally. Such a finding reveals that most LLM are undertrained for their size— _we would benefit from training existing LLMs over significantly more data_. For example, the scaling laws fit in [6] predict that Gopher should have been trained on a 20x larger dataset!\n> _“The amount of training data that is projected to be needed is far beyond what is currently used to train large models.”_ - from [6]\n**Chinchilla.** The analysis provided in [6] emphasizes the importance of data scale. _Large models need to be trained over more data to reach their best performance_. To validate this finding, authors train a 70 billion parameter LLM, called Chinchilla. Compared to prior models, Chinchilla is smaller but has a larger pretraining dataset— _1.4T training tokens in total_. Chinchilla uses the same data and evaluation strategy as Gopher [10]. Despite being 4x smaller than Gopher, Chinchilla consistently outperforms the larger model; see below.\n[![](https://substackcdn.com/image/fetch/$s_!gWSa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F248efee3-31f7-4b53-9709-270254e13547_1138x1394.png)](https://substackcdn.com/image/fetch/$s_!gWSa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F248efee3-31f7-4b53-9709-270254e13547_1138x1394.png)(from [6])\nScaling laws proposed by Chinchilla [6] became a standard in AI research for years afterward. _“Chinchilla-optimal”_ is now a commonly-used term. Even today, after a wide variety of additional scaling research has been published, Chinchilla and its associated scaling laws are referenced constantly.\n## The “Death” of Scaling Laws\nScaling laws have recently become a popular (and contentious) topic within AI research. As we have seen throughout this overview, scaling has fueled most improvements in AI throughout the age of pretraining. As the pace of model releases and improvements slowed in the latter half of 2024[11](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-11), however, we began to see widespread questioning of model scaling, seeming to indicate that AI research— _and scaling laws in particular_ —could be hitting a wall.\n  * [Reuters](https://www.reuters.com/technology/artificial-intelligence/openai-rivals-seek-new-path-smarter-ai-current-methods-hit-limitations-2024-11-11) claims that OpenAI is shifting their product strategy due to hitting a plateau in scaling current methods. \n  * [The Information](https://www.theinformation.com/articles/openai-shifts-strategy-as-rate-of-gpt-ai-improvements-slows) claims that the rate of improvement in GPT models is starting to slow down. \n  * [Bloomberg](https://www.bloomberg.com/news/articles/2024-11-13/openai-google-and-anthropic-are-struggling-to-build-more-advanced-ai) highlights the difficulties being faced by several frontier labs in trying to build AI that is more capable.\n  * [TechCrunch](https://techcrunch.com/2024/11/20/ai-scaling-laws-are-showing-diminishing-returns-forcing-ai-labs-to-change-course/) claims that scaling is starting to yield diminishing returns. \n  * [Time](https://time.com/7178328/is-ai-progress-slowing-down/) publishes a nuanced essay that highlights various contributing factors to the narrative that AI research is slowing down. \n  * Ilya Sutskever states that _“pretraining as we know it will end”_ during his [test of time award speech](https://www.youtube.com/watch?v=1yvBqasHLZs) at NeurIPS’24. \n\n\nAt the same time, many experts are arguing the opposite. For example, [Dario Amodei](https://www.youtube.com/watch?v=ugvHCXCOmm4) (Anthropic CEO) has said that scaling is _“probably… going to continue”_ , while Sam Altman has continued pushing the narrative that _[“there is no wall”](https://x.com/sama/status/1856941766915641580)_. In this section, we will add more color to this discussion by providing a grounded explanation of the current state of scaling and various issues that may exist.\n#### Slower Scaling: What does it mean? Why is it happening?\n> _“Both narratives can be true: Scaling is still working at a technical level. The rate of improvement for users is slowing.”_ - [Nathan Lambert](https://www.interconnects.ai/p/scaling-realities)\nSo… _is scaling slowing down?_ The answer is complex and highly dependent upon our exact definition of “slowing down”. So far, the most reasonable response that I’ve seen to this question is that both answers are correct. For this reason, we will not try to answer this question. Instead, we will simply take a deeper look at what exactly the research tells us about this topic so that we can build a more nuanced understanding of the current (and future) state of scaling for LLMs. \n**What do scaling laws tell us?** First, we need to recall the technical definition of a scaling law. Scaling laws define a relationship, based upon a power law, between training compute (or model / dataset size) and the test loss of an LLM. However, _the nature of this relationship is often misunderstood_. The idea of getting exponential performance improvements from logarithmic increases in compute is a myth. Scaling laws look more like an exponential decay, meaning that we will have to work harder over time to get further performance improvements; see below.\n[![](https://substackcdn.com/image/fetch/$s_!NVCM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95f0364e-8bdb-4144-8ff0-cb6c723af58e_1098x708.png)](https://substackcdn.com/image/fetch/$s_!NVCM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95f0364e-8bdb-4144-8ff0-cb6c723af58e_1098x708.png)(from [5])\nIn other words, _scaling laws plateau naturally over time_. In this way, the “slowdown” we are currently experiencing is arguably an expected part of LLM scaling laws.\n> _“Practitioners often use downstream benchmark accuracy as a proxy for model quality and not loss on perplexity evaluation sets.”_ - from [7]\n**Defining performance.**_How do we measure whether LLMs are improving or not?_ From the perspective of scaling laws, LLM performance is usually measured via the model’s test loss during pretraining, but the impact of lower test loss on an LLM’s capabilities is unclear. _Does lower loss lead to higher accuracy on downstream tasks? Does lower loss cause an LLM to acquire new capabilities?_ There is a disconnect here between what scaling laws tell us and what we actually care about:\n  * Scaling laws tell us that increasing the scale of pretraining will smoothly decrease the LLM’s test loss.\n  * We care about getting a “better” LLM.\n\n\nDepending on who you are, your expectations of new AI systems— _and the approach that you use to evaluate these new systems_ —will vary drastically. Average AI users tend to focus upon general chat applications, while practitioners often care about an LLM’s performance on downstream tasks. In contrast, researchers at top frontier labs seem to have high (and very particular) expectations of AI systems; e.g., [writing a Ph.D. thesis](https://time.com/7178328/is-ai-progress-slowing-down/) or solving [advanced mathematical reasoning problems](https://epoch.ai/frontiermath/the-benchmark). Given that LLMs have such broad capabilities, evaluation is difficult, and there are many lenses from which we can view an LLM’s performance; see below. \n[![](https://substackcdn.com/image/fetch/$s_!L4Aa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a11f01c-02c4-4ede-bf3f-cbda3f5df9a5_1250x840.png)](https://substackcdn.com/image/fetch/$s_!L4Aa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a11f01c-02c4-4ede-bf3f-cbda3f5df9a5_1250x840.png)(from [15])\nGiven this drastic variance in model expectations, providing definitive proof that scaling is “working” will always be a struggle. We need a more specific definition of success for scaling laws. If the science tells us that larger models will achieve lower loss, this does not mean that new models will meet everyone’s expectations. A failure to achieve AGI or surpass the capabilities of award-winning human mathematicians is not proof that scaling is not still working at a technical level! Put differently, one could argue that the “slowdown” of scaling is a perception and expectation problem rather than a technical problem with scaling laws.\n**Data death.** To scale up LLM pretraining, we must increase both model and dataset size. Earlier research [1] seemed to indicate that the amount of data was not as important as the size of the model, but we see with Chinchilla [6] that dataset size is equally important. Plus, more recent work argues that most researchers prefer to “overtrain” their models— _or pretrain them on datasets that go beyond Chinchilla-optimality in size_ —to save on inference costs [7].\n> _“Scaling studies usually focus on the compute-optimal training regime... As larger models are more expensive at inference, it is now common practice to over-train smaller models.”_ - from [7]\nAll of this research brings us to one simple conclusion— _scaling up LLM pretraining will require that we create larger pretraining datasets_. This fact forms the basis for one of the key criticisms of LLM scaling laws. Many researchers believe that there may not be enough data available to continue scaling the pretraining process. For context, a large majority of pretraining data used for current LLMs is obtained via web scraping; see below. Given that we only have one internet, finding completely new sources of large-scale, high-quality pretraining data may be hard. \n[![](https://substackcdn.com/image/fetch/$s_!GgSm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3fe055b-9ca4-410c-b36d-d6e8047b4b28_1554x924.png)](https://substackcdn.com/image/fetch/$s_!GgSm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3fe055b-9ca4-410c-b36d-d6e8047b4b28_1554x924.png)([source](https://docs.google.com/presentation/d/179dpzWSQ9G7EAUlvaJdeE0av9PLuk9Rl33nfhHSJ4xI/edit#slide=id.g31d042f750e_0_808))\nEven Ilya Sutskever has [recently made this argument](https://www.reuters.com/technology/artificial-intelligence/ai-with-reasoning-power-will-be-less-predictable-ilya-sutskever-says-2024-12-14/), claiming that _i)_ compute is growing quickly but _ii)_ data is not growing due to reliance upon web scraping. Therefore, he believes that we cannot continue scaling up the pretraining process forever. Pretraining as we know it will end, and we must find new avenues of progress for AI research. In other words, _“we have achieved peak data”_. \n#### The Next Generation of Scale for Pretraining\nScaling will eventually lead to diminishing returns, and the data-centric argument against the continuation of scaling is both sound and compelling. However, there are still several directions of research that could improve the pretraining process.\n[![](https://substackcdn.com/image/fetch/$s_!vWyx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81237158-d1dd-4f39-a77b-ea3f70f83a6d_1756x970.png)](https://substackcdn.com/image/fetch/$s_!vWyx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81237158-d1dd-4f39-a77b-ea3f70f83a6d_1756x970.png)\n**Synthetic data.** To scale up the pretraining process several orders of magnitude, we will likely need to rely upon synthetically-generated data. Despite worries that over reliance upon synthetic data will lead to diversity issues [14], we are seeing increased— _and[seemingly successful](https://www.interconnects.ai/p/llm-synthetic-data)_—usage of synthetic data for LLMs [12]. Additionally, curriculum learning [13] and [continued pretraining](https://www.databricks.com/blog/characterizing-datasets-and-building-better-models-continued-pre-training) strategies have led to a variety of meaningful improvements via adjustments to pretraining data; e.g., changing data mixtures or adding instruction data at the end of pretraining. \n[![](https://substackcdn.com/image/fetch/$s_!kRSV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe851103a-aa40-47cf-b2cc-85f8cee44e06_1274x910.png)](https://substackcdn.com/image/fetch/$s_!kRSV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe851103a-aa40-47cf-b2cc-85f8cee44e06_1274x910.png)(from [7])\n**Practical scaling laws.** Recent research has attempted to address the limitations of test loss-based scaling laws. For example, authors in [7] define scaling laws that can be used to predict the performance of an LLM on downstream benchmarks from the [LLM Foundry](https://github.com/mosaicml/llm-foundry); see above. Interpreting these kinds of metrics is much easier for humans. We might not know what a 5% decrease in test loss means, but jumping from 85% to 90% accuracy on our benchmark of interest is usually easy to grasp. Several other works have also explored the idea of using scaling laws to provide more practical and meaningful estimates of LLM performance; e.g., after post training and quantization [16] or during the pretraining process [17].\n**DeepSeek-v3.** Despite recent arguments, we are still seeing semi-frequent advancements by scaling the LLM pretraining process. For example, DeepSeek-v3 [18]— _a 671B parameter_[12](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-12) _[mixture-of-experts](https://cameronrwolfe.substack.com/p/conditional-computation-the-birth) (MoE) model_—was recently released. In addition to being open-source, this model is pretrained on 14.8T tokens of text and surpasses the performance of GPT-4o and Claude-3.5-Sonnet; see below for the model’s performance and [here](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/LICENSE-MODEL) for the license. For reference, LLaMA-3 models are trained on over 15T of raw text data; see [here](https://ai.meta.com/blog/meta-llama-3/) for more details. \n[![](https://substackcdn.com/image/fetch/$s_!NntB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbe15be-8b38-4584-bba9-c9c5e20c5806_1586x1018.png)](https://substackcdn.com/image/fetch/$s_!NntB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbe15be-8b38-4584-bba9-c9c5e20c5806_1586x1018.png)(from [18])\nThe ability to outperform models like GPT-4o is a significant jump for open weights LLMs— _even the[largest LLaMA models](https://ai.meta.com/blog/meta-llama-3-1/) have fallen short of this goal_. DeepSeek-v3 adopts a variety of interesting tricks:\n  * An optimized MoE architecture from [DeepSeek-v2](https://arxiv.org/abs/2405.04434). \n  * A new auxiliary-loss-free strategy for [load balancing](https://cameronrwolfe.substack.com/i/142423094/switch-transformers-scaling-to-trillion-parameter-models-with-simple-and-efficient-sparsity) the MoE.\n  * A [multi-token prediction](https://arxiv.org/abs/2404.19737) training objective.\n  * Distillation of reasoning capabilities from a long-chain-of-thought model (i.e., similar to [OpenAI’s o1](https://openai.com/index/learning-to-reason-with-llms/)).\n\n\nThe model also undergoes post-training, including supervised finetuning and reinforcement learning from human feedback, to align it to human preferences. \n> _“We train DeepSeek-V3 on 14.8T high-quality and diverse tokens. The pre-training process is remarkably stable. Throughout the entire training process, we did not encounter any irrecoverable loss spikes or have to roll back.”_ - from [8]\nHowever, the biggest key to DeepSeek-v3’s impressive performance is pretraining scale— _this is a massive model trained over an equally-massive dataset_! Training such a large model is difficult for a variety of reasons (e.g., GPU failures and loss spikes). DeepSeek-v3 has a surprisingly stable pretraining process and is trained at a reasonable cost by LLM standards; see below. _These results indicate that larger-scale pretraining jobs are becoming more manageable and efficient over time_.\n[![](https://substackcdn.com/image/fetch/$s_!bO6X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f81e548-b371-4576-a694-a50748f5f148_1586x288.png)](https://substackcdn.com/image/fetch/$s_!bO6X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f81e548-b371-4576-a694-a50748f5f148_1586x288.png)(from [18])\n**Increasing scale by an OOM.** To continue testing our scaling laws, we must train LLMs that are several orders of magnitude beyond current models. Setting aside our opinions on the utility of scaling, there are still a variety of limitations that stand in the way of training models of this scale. We will need:\n  * Larger compute clusters[13](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-13). \n  * More (and better) hardware.\n  * Massive amounts of power.\n  * New algorithms (e.g., for larger-scale distributed training, [potentially spanning multiple data centers](https://www.youtube.com/watch?v=pE3KKUKXcTM)).\n\n\nTraining the next generations of models is not just a matter of securing funding for more GPUs, it is a multi-disciplinary feat of engineering. Such complex efforts take time. For reference, GPT-4 was released in March of 2023, nearly three years— _33 months in particular_ —after the release of GPT-3. It is reasonable to expect a similar timeline (if not longer) for unlocking another 10-100x increase in scale.\n> _“At every order of magnitude scale up, different innovations have to be found.”_ - [Ege Erdil](https://scholar.google.com/citations?user=x4aaIwcAAAAJ&hl=en) ([Epoch AI](https://epoch.ai/))\n## The Future(s) of AI Research\nNow that we more deeply understand the state of scaling for pretraining, let’s assume (purely for the purpose of discussion) that pretraining research will hit a sudden wall. Even if model capabilities do not improve at all in the near future, there are a variety of ways in which AI research can continue to rapidly progress. We have already talked about some of these topics (e.g., synthetic data). In this section, we will focus on two topics in particular that are currently popular:\n  1. LLM systems / agents.\n  2. Reasoning models.\n\n\n#### Building Useful LLM Systems\nMost LLM-based applications today operate in a single-model paradigm. In other words, we solve tasks by passing that task to a single LLM and directly using the model’s output as the answer to that task; see below.\n[![](https://substackcdn.com/image/fetch/$s_!A46L!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c6aaa13-cbb4-45e9-baa7-f65e6f176538_1014x976.png)](https://substackcdn.com/image/fetch/$s_!A46L!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c6aaa13-cbb4-45e9-baa7-f65e6f176538_1014x976.png)\nIf we want to improve such a system (i.e., solve harder tasks with higher accuracy), we could simply improve the capabilities of the underlying model, but such an approach is dependent upon the creation of more capable models. Instead, we could go beyond the single-model paradigm by building an LLM-based system that combines several LLMs— _or other components_ —to solve a complex task. \n[![](https://substackcdn.com/image/fetch/$s_!QxDk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F775634bb-6519-4495-a3b1-83a437b590de_2240x782.png)](https://substackcdn.com/image/fetch/$s_!QxDk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F775634bb-6519-4495-a3b1-83a437b590de_2240x782.png)\n**LLM system basics.** The goal of an LLM system is to break complex tasks into smaller parts that are easier to solve for an LLM or other module. There are two primary strategies we can use to accomplish this goal (depicted above):\n  1. _Task decomposition_ : break the task itself into smaller sub-tasks that can be solved individually and aggregated[14](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-14)afterwards to form a final answer.\n  2. _Chaining_ : solve a task or sub-task by making several sequential calls to an LLM instead of a single call.\n\n\nThese strategies can be used alone or in tandem. For example, assume that we want to build a system to summarize books. To do this, we could decompose the task by first summarizing each chapter of the book. From here, we can either:\n  * Further decompose the task into smaller chunks of text to summarize (i.e., similar to [recursive / hierarchical decomposition](https://openai.com/index/summarizing-books/)).\n  * Chain several LLM calls together; e.g., have one LLM extract all important facts or information from the chapter and another LLM produce the chapter summary based on these key facts.\n\n\nThen, we can aggregate these results by asking an LLM to summarize the concatenated chapter summaries, forming a summary of the full novel. Most complex tasks can be decomposed into simple parts that are easy to solve, which makes such LLM systems very powerful. These systems can become incredibly sophisticated as we perform more extensive decomposition and chaining, making them an interesting (and impactful) area of applied AI research. \n**Building LLM-based products.** Despite the success and popularity of LLMs, the number of practical (and widely-adopted) use cases for LLMs is still very small. The largest use cases for LLMs today are code generation and chat, both of which are relatively obvious applications for LLMs[15](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-15); see below.\n[![](https://substackcdn.com/image/fetch/$s_!XmzO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb4408e7-7a56-4ffb-8561-e8261dda1022_2560x1258.png)](https://substackcdn.com/image/fetch/$s_!XmzO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb4408e7-7a56-4ffb-8561-e8261dda1022_2560x1258.png)([source](https://menlovc.com/2024-the-state-of-generative-ai-in-the-enterprise/))\nGiven that there are so many ripe domains for the application of LLMs, _simply building more LLM-based products that are genuinely useful is an important area of applied AI research_. We already have access to very capable models, but using these models to build a product worth using is an entirely different problem. Solving this problem requires learning how to build reliable and capable LLM systems.\n[![](https://substackcdn.com/image/fetch/$s_!0EW2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd1a733fb-4558-4ec4-8aa8-f130199a646c_1788x1212.png)](https://substackcdn.com/image/fetch/$s_!0EW2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd1a733fb-4558-4ec4-8aa8-f130199a646c_1788x1212.png)(from [19])\n**Agents.** The line between LLM systems and agents is blurry because the term “agent” has been [overloaded](https://www.interconnects.ai/p/the-ai-agent-spectrum) by the AI community. However, the key concept that we should understand is that _LLM systems can be expanded in a variety or interesting and meaningful ways_. For example, we can augment LLMs by teaching them how to use tools (e.g., calculators, search engines, etc.) when solving a problem; see above. Additionally, we can allow LLMs to execute their own programs or even perform actions for us; e.g., booking a hotel or sending an email. The many modules and tools that can be integrated with an LLM present endless possibilities for building LLM systems that are more capable and useful.\n**Robustness** is one of the biggest roadblocks to building more capable LLM / agent systems. Assume that we have an LLM system that makes ten different calls to an LLM. Additionally, let’s assume that each of these LLM calls have a 95% likelihood of success and that all of the calls need to succeed in order to generate the correct final outputs. Although the individual components of this system are reasonably accurate, _the overall system has a success rate of 60%_!\n[![](https://substackcdn.com/image/fetch/$s_!B-VP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e7871ad-0e65-4b86-a7bb-3a7b62566e4f_1274x294.png)](https://substackcdn.com/image/fetch/$s_!B-VP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e7871ad-0e65-4b86-a7bb-3a7b62566e4f_1274x294.png)(from [20])\nAs we add more components, this problem becomes exponentially worse, which limits the complexity of LLM / agent systems that we can build. Building more complex systems will require that we drastically improve the robustness of each individual system component. [Recent research](https://www.interconnects.ai/p/how-scaling-changes-model-behavior) indicates that robustness can be improved via scaling. However, we can also improve robustness via better meta-generation algorithms; see above. Instead of generating a single output from an LLM, these algorithms use ideas like parallel decoding, (step-level) verification, critiques, and more to obtain a more refined and accurate output from an LLM.\n[![](https://substackcdn.com/image/fetch/$s_!QMa3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafb7f02e-15bf-4445-aec6-f98dd061e36a_1276x726.png)](https://substackcdn.com/image/fetch/$s_!QMa3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafb7f02e-15bf-4445-aec6-f98dd061e36a_1276x726.png)(from [20])\nThis area of research is rapidly progressing and is likely to become a key driver of progress in AI research; see [20] for an in-depth survey of the topic. As meta-generation algorithms improve, LLMs will become more robust, and we will become capable of building increasingly complex LLM / agent systems. \n#### Reasoning Models and New Scaling Paradigms\nA common criticism of early LLMs was that they simply memorized data and had very little reasoning capabilities. However, the inability of LLMs to reason has been largely debunked over the last few years. We have learned from recent research that these models likely had the inherent ability to reason all along, but we have to use the correct prompting or training approach to elicit this ability.\n**Chain of thought (CoT) prompting** [22] was one of the first techniques to demonstrate the reasoning capabilities of LLMs. The approach is simple and prompt-based. We just ask the LLM to provide an explanation for its response prior to generating the actual response; see [here](https://cameronrwolfe.substack.com/p/chain-of-thought-prompting-for-llms) for more details. When an LLM generates a rationale that outlines the [step-by-step](https://arxiv.org/abs/2205.11916) process used to arrive at a response, its reasoning capabilities improve significantly. Plus, this explanation is human-readable and can make the model’s output more interpretable!\n[![](https://substackcdn.com/image/fetch/$s_!NPw_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F599a636e-b0b2-4de3-84c8-3edf906bfa82_1616x882.png)](https://substackcdn.com/image/fetch/$s_!NPw_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F599a636e-b0b2-4de3-84c8-3edf906bfa82_1616x882.png)(from [22])\nThe idea of a chain of thought is both generic and powerful. In fact, chains of thought have become the key concept behind improving LLM reasoning capabilities, and we have seen this technique repurposed in many ways:\n  * LLM-as-a-Judge-style evaluation models usually provide scoring rationales prior to generating a final evaluation result [23, 24].\n  * Supervised finetuning and instruction tuning strategies have been proposed for teaching smaller / open LLMs to write better chains of thought [25, 26]. \n  * LLMs are commonly asked to reflect upon and critique or verify their own outputs, then revise their output based on this information [12, 27]. \n\n\nComplex reasoning is an active research topic that is quickly progressing. New training algorithms that teach LLMs to incorporate (step-level) verification [28, 29] into their reasoning process have shown promising results, and we will likely continue to see improvements as new and better training strategies emerge. \n**OpenAI’s o1 reasoning model**[21] marks a significant jump in the reasoning capabilities of LLMs. The reasoning strategy used by o1 is heavily based upon chains of thought. Similar to how humans think before responding to a question, o1 will spend time “thinking” before providing a response. Practically speaking, the “thoughts” that o1 generates are just long chains of thought that the model uses to think through a problem, break the problem into simpler steps, try various approaches to solving the problem, and even correct its own mistakes[16](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-16).\n> _“OpenAI o1 [is] a new large language model trained with RL to perform complex reasoning. o1 thinks before it answers—it can produce a long internal chain of thought before responding to the user.”_ - from [21]\nThe details of o1’s exact training strategy are not publicly shared. However, we know that o1 is taught to reason using a “large-scale reinforcement learning” algorithm that is “highly data-efficient” and focuses upon refining the model’s ability to generate useful chains of thought. Based upon [public comments](https://x.com/__nmca__/status/1870170098989674833) from OpenAI researchers and [recent rhetoric](https://www.interconnects.ai/p/openais-o1-using-search-was-a-psyop) regarding o1, the model seems to be trained using pure reinforcement learning, which contradicts earlier opinions that o1 may be using some form of tree search at inference time. \n[![Breakdown of the accuracy and raw score of gpt-4o vs. o1 on various competition evals](https://substackcdn.com/image/fetch/$s_!9A6I!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F896f5300-4bfe-4595-80e7-1063b4225a5b_2400x1650.png)](https://substackcdn.com/image/fetch/$s_!9A6I!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F896f5300-4bfe-4595-80e7-1063b4225a5b_2400x1650.png)Comparison between GPT-4o and o1 on reasoning-heavy tasks (from [21]) \nAs mentioned, the performance of o1 on complex reasoning tasks is impressive. o1 outperforms GPT-4o on nearly all reasoning-heavy tasks; see above. As an example of o1’s reasoning capabilities, the model:\n  * Places in the 89th percentile on competitive programming questions from [Codeforces](https://codeforces.com/).\n  * Reaches the top 500 students in the US in a qualifier for the USA Math Olympiad ([AIME](https://artofproblemsolving.com/wiki/index.php/American_Invitational_Mathematics_Examination?srsltid=AfmBOoplGn5Sl-tAVYpjGeZiZChoPCs-y6Ofqk90mzlvesrqhkA5kL6s)).\n  * Exceeds the accuracy of human PhD candidates on graduate-level physics, biology and chemistry problems ([GPQA](https://arxiv.org/abs/2311.12022)).\n\n\n[![The image shows two scatter plots comparing \"o1 AIME accuracy\" during training and at test time. Both charts have \"pass@1 accuracy\" on the y-axis and compute \\(log scale\\) on the x-axis. The dots indicate increasing accuracy with more compute time.](https://substackcdn.com/image/fetch/$s_!OAWu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9522717b-cade-49a2-a2e4-f60eb1007127_1980x1113.webp)](https://substackcdn.com/image/fetch/$s_!OAWu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9522717b-cade-49a2-a2e4-f60eb1007127_1980x1113.webp)(from [22])\n**From o1 to o3.** One of the most interesting aspects of o1 is that the model’s reasoning capabilities can be improved by using more compute at inference time. To solve problems of increasing complexity, the model can simply generate progressively longer chains of thought; see [here](https://openai.com/index/learning-to-reason-with-llms/) for examples. Using more inference-time compute to generate these longer chains of thought yields a smooth increase in the model’s reasoning performance; see below.\n> _“We have found that the performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute).”_ - from [22]\nSimilarly, we see in the figure above that o1’s performance improves smoothly as we invest more compute into training via reinforcement learning. This is exactly the approach that was followed to create the o3 reasoning model. The evaluation results for this model were [previewed](https://www.youtube.com/watch?v=SKBG1sqdyIU&t=2s) by OpenAI in late 2024, and very few details about o3 have been publicly shared. However, given that this model was released so quickly after o1 (i.e., three months later), o3 is most likely a [“scaled up” version of o1](https://www.interconnects.ai/p/openais-o3-the-2024-finale-of-ai) that invests more compute into reinforcement learning.\n[![](https://substackcdn.com/image/fetch/$s_!iCwS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2975bbda-3d86-4cf6-9108-9c20abc9ec59_1286x1300.png)](https://substackcdn.com/image/fetch/$s_!iCwS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2975bbda-3d86-4cf6-9108-9c20abc9ec59_1286x1300.png)([source](https://arcprize.org/blog/oai-o3-pub-breakthrough))\nThe o3 model has not yet been released at the time of writing, but the results achieved by scaling o1 are incredibly impressive (even shocking in some cases). The most notable achievements of o3 are listed below:\n  * A score of 87.5% on the [ARC-AGI benchmark](https://arcprize.org/blog/oai-o3-pub-breakthrough) on which GPT-4o achieves only 5% accuracy. o3 is the first model to exceed human-level performance of 85% on ARC-AGI. This benchmark has been described as a “North Star” towards AGI and has remained unbeaten[17](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-17) for over five years. \n  * An accuracy of 71.7% on [SWE-Bench Verified](https://openai.com/index/introducing-swe-bench-verified/) and an [Elo score](https://en.wikipedia.org/wiki/Elo_rating_system) of 2727 on Codeforces, which ranks o3 among the top 200 human competitive programmers on the planet.\n  * An accuracy of 25.2% on EpochAI’s [FrontierMath benchmark](https://epoch.ai/frontiermath), _improving upon the previous state-of-the-art accuracy of 2.0%_. This benchmark was described by [Terence Tao](https://en.wikipedia.org/wiki/Terence_Tao) as “incredibly difficult” and likely to be unsolved by AI systems for “several years at least”. \n\n\nA distilled version of o3, called o3-mini, was also previewed, which performs very well and comes with significant improvements in compute efficiency.\n[![](https://substackcdn.com/image/fetch/$s_!OfaP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1e98bbb-e1d6-454b-ac9f-c43a38ba0fb5_2516x952.png)](https://substackcdn.com/image/fetch/$s_!OfaP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1e98bbb-e1d6-454b-ac9f-c43a38ba0fb5_2516x952.png)(from [21] and [here](https://arcprize.org/blog/oai-o3-pub-breakthrough))\n**New paradigms for scaling.** After reading this overview, many of the plots presented by o1 and o3 (see above) might look pretty familiar— _these are log-scale plots where we see a smooth, linear increase in performance with more compute_! In other words, we see a clear power law relationship between the performance of these reasoning models and two different quantities:\n  * Training-time (reinforcement learning) compute. \n  * Inference-time compute.\n\n\nScaling o1-style models differs from traditional scaling laws. Instead of scaling up the pretraining process, we are scaling the amount of compute invested into post training and inference. _This is a completely new scaling paradigm_ , and the results achieved by scaling reasoning models are great so far. Such a finding shows us that other avenues of scaling— _beyond pretraining_ —clearly exist. With the advent of reasoning models, we have discovered our next hill to climb. Although it may come in a different form, _scaling will continue to drive progress in AI research_.\n## Closing Thoughts\nWe now have a clearer view of scaling laws, their impact on LLMs, and the future directions of progress for AI research. As we have learned, there are many contributing factors to the recent criticism of scaling laws:\n  * The natural decay in scaling laws.\n  * The high variance in expectations of LLM capabilities.\n  * The latency of large-scale, inter-disciplinary engineering efforts.\n\n\nThese issues are valid, _but_ _none of them indicate that scaling is not still working as expected_. Investments into large-scale pretraining will (and should) continue, but improvements will become exponentially harder over time. As a result, alternative directions of progress (e.g., agents and reasoning) will become more important. As we invest into these new areas of research, however, the fundamental idea of scaling will continue to play a massive role. Whether scaling will continue is not a question. _The true question is what we will scale next_.\n#### New to the newsletter?\nHi! I’m [Cameron R. Wolfe](https://cameronrwolfe.me/), Deep Learning Ph.D. and Machine Learning Scientist at [Netflix](https://research.netflix.com/research-area/nlp-and-conversations). This is the Deep (Learning) Focus newsletter, where I help readers better understand important topics in AI research. If you like the newsletter, please subscribe, share it, or follow me on [X](https://twitter.com/cwolferesearch) and [LinkedIn](https://www.linkedin.com/in/cameron-r-wolfe-ph-d-04744a238/)!\nSubscribe\n#### Bibliography\n[1] Kaplan, Jared, et al. \"Scaling laws for neural language models.\" _arXiv preprint arXiv:2001.08361_ (2020).\n[2] Radford, Alec. \"Improving language understanding by generative pre-training.\" (2018).\n[3] Radford, Alec, et al. \"Language models are unsupervised multitask learners.\" _OpenAI blog_ 1.8 (2019): 9.\n[4] Brown, Tom, et al. \"Language models are few-shot learners.\" _Advances in neural information processing systems_ 33 (2020): 1877-1901.\n[5] Achiam, Josh, et al. \"Gpt-4 technical report.\" _arXiv preprint arXiv:2303.08774_ (2023).\n[6] Hoffmann, Jordan, et al. \"Training compute-optimal large language models.\" _arXiv preprint arXiv:2203.15556_ (2022).\n[7] Gadre, Samir Yitzhak, et al. \"Language models scale reliably with over-training and on downstream tasks.\" _arXiv preprint arXiv:2403.08540_ (2024).\n[8] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" _Advances in neural information processing systems_ 35 (2022): 27730-27744.\n[9] Smith, Shaden, et al. \"Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model.\" _arXiv preprint arXiv:2201.11990_ (2022).\n[10] Rae, Jack W., et al. \"Scaling language models: Methods, analysis & insights from training gopher.\" _arXiv preprint arXiv:2112.11446_ (2021).\n[11] Bhagia, Akshita, et al. \"Establishing Task Scaling Laws via Compute-Efficient Model Ladders.\" _arXiv preprint arXiv:2412.04403_ (2024).\n[12] Bai, Yuntao, et al. \"Constitutional ai: Harmlessness from ai feedback.\" _arXiv preprint arXiv:2212.08073_ (2022).\n[13] Blakeney, Cody, et al. \"Does your data spark joy? Performance gains from domain upsampling at the end of training.\" _arXiv preprint arXiv:2406.03476_ (2024).\n[14] Chen, Hao, et al. \"On the Diversity of Synthetic Data and its Impact on Training Large Language Models.\" _arXiv preprint arXiv:2410.15226_ (2024).\n[15] Guo, Zishan, et al. \"Evaluating large language models: A comprehensive survey.\" _arXiv preprint arXiv:2310.19736_ (2023).\n[16] Xu, Zifei, et al. \"Scaling laws for post-training quantized large language models.\" _arXiv preprint arXiv:2410.12119_ (2024).\n[17] Xiong, Yizhe, et al. \"Temporal scaling law for large language models.\" _arXiv preprint arXiv:2404.17785_ (2024).\n[18] DeepSeek-AI et al. \"DeepSeek-v3 Technical Report.\" _https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf_ (2024).\n[19] Schick, Timo, et al. \"Toolformer: Language models can teach themselves to use tools.\" arXiv preprint arXiv:2302.04761 (2023).\n[20] Welleck, Sean, et al. \"From decoding to meta-generation: Inference-time algorithms for large language models.\" _arXiv preprint arXiv:2406.16838_ (2024).\n[21] OpenAI et al. “Learning to Reason with LLMs.” _https://openai.com/index/learning-to-reason-with-llms/_ (2024).\n[22] Wei, Jason, et al. \"Chain-of-thought prompting elicits reasoning in large language models.\" _Advances in neural information processing systems_ 35 (2022): 24824-24837.\n[23] Liu, Yang, et al. \"G-eval: Nlg evaluation using gpt-4 with better human alignment.\" _arXiv preprint arXiv:2303.16634_ (2023).\n[24] Kim, Seungone, et al. \"Prometheus: Inducing fine-grained evaluation capability in language models.\" _The Twelfth International Conference on Learning Representations_. 2023.\n[25] Ho, Namgyu, Laura Schmid, and Se-Young Yun. \"Large language models are reasoning teachers.\" _arXiv preprint arXiv:2212.10071_ (2022).\n[26] Kim, Seungone, et al. \"The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning.\" _arXiv preprint arXiv:2305.14045_ (2023).\n[27] Weng, Yixuan, et al. \"Large language models are better reasoners with self-verification.\" _arXiv preprint arXiv:2212.09561_ (2022).\n[28] Lightman, Hunter, et al. \"Let's verify step by step.\" _arXiv preprint arXiv:2305.20050_ (2023).\n[29] Zhang, Lunjun, et al. \"Generative verifiers: Reward modeling as next-token prediction.\" _arXiv preprint arXiv:2408.15240_ (2024).\n[1](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-1)\nThe two primary reports are from [The Information](https://www.theinformation.com/articles/openai-shifts-strategy-as-rate-of-gpt-ai-improvements-slows) and [Reuters](https://www.reuters.com/technology/artificial-intelligence/openai-rivals-seek-new-path-smarter-ai-current-methods-hit-limitations-2024-11-11/). \n[2](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-2)\nWe use the following settings to generate the plot: `a = 1`, `p = 0.5`, and `0 < x < 1`.\n[3](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-3)\nCompute is defined in [1] as `6NBS`, where `N` is the number of model parameters, `B` is the batch size used during training, and `S` is the total number of training steps. \n[4](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-4)\nThis extra multiplicative constant does not change the behavior of the power law. To understand why this is the case, we must understand the definition of [scale invariance](http://felix.physics.sunysb.edu/~allen/540-05/scaling.html#:~:text=scale%20invariance%20of%20power%20law%20functions&text=The%20function%20y%3Dxp,by%20a%20scale%20factor%20a.). Because power laws are scale invariant, the fundamental characteristics of a power law are the same even when we scale up or down by a certain factor. The behavior observed will be the same at any scale!\n[5](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-5)\nThis description was from Ilya’s test of time award for [this paper](https://arxiv.org/abs/1409.3215) at NeurIPS’24. \n[6](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-6)\nAlthough this might seem obvious now, we should remember that most NLP tasks at the time (e.g., summarization and QA) had entire areas of research devoted to them! Each of these tasks had associated task-specific architectures that were specialized in performing that task, and GPT was a single generic model that could outperform most of these architectures across several different tasks. \n[7](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-7)\nThis means that we just describe each task in the LLM’s prompt and use the same model to solve different tasks— _only the prompt changes between tasks_.\n[8](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-8)\nThis is to be expected because these models use zero-shot inference and are not finetuned at all on any of the downstream tasks. \n[9](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-9)\nBy an “emergent” ability, we mean a skill possessed by the LLM that only emerges after a certain scale has been reached (e.g., a sufficiently large model). \n[10](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-10)\nHere, we define “compute-optimal” as the training setting that yields the best possible performance— _in terms of test loss_ —at a fixed training compute cost. \nbest performance at a fixed compute cost\n[11](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-11)\nFor example, Anthropic has continually pushed back their release of [Claude 3.5 Opus](https://www.anthropic.com/news/claude-3-5-sonnet), Google has only released the [flash variant of Gemini-2](https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/), and OpenAI only released [GPT-4o](https://openai.com/index/hello-gpt-4o/) in 2024 (until [o1](https://openai.com/o1/) and o3 were released in December), which was arguably not significantly more capable than GPT-4. \n[12](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-12)\nOnly 37 billion of these parameters are active during inference for a single token. \n[13](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-13)\nFor example, xAI has recently built a [new datacenter](https://www.datacenterfrontier.com/machine-learning/article/55244139/the-colossus-ai-supercomputer-elon-musks-drive-toward-data-center-ai-technology-domination) with 100,000 NVIDIA GPUs in Memphis, while Anthropic leadership has [voiced a desire](https://www.youtube.com/watch?v=ugvHCXCOmm4&t=1103s) to increase their compute spend by up to 100x over the next few years. \n[14](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-14)\nThe aggregation step can be implemented in a variety of ways. For example, we can manually aggregate responses (e.g., via concatenation), use an LLM, or pretty much anything in between!\n[15](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-15)\nThis is not because these tasks are simple. Both code generation and chat are hard to solve, but they are (arguably) fairly obvious applications for LLMs. \n[16](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-16)\nOpenAI has chosen to hide these long chains of thought from the users of o1. The argument behind this choice is that these rationales provide insight into the model’s though process that can be used to debug or monitor the model. However, the model should be allowed to express its pure thoughts without any of the safety filtering that is necessary for user-facing model outputs. \n[17](https://cameronrwolfe.substack.com/p/llm-scaling-laws#footnote-anchor-17)\nCurrently, ARC-AGI is technically still unbeaten because o3 exceeds the compute requirement for the benchmark. However, the model still achieves an accuracy of 75.7% using a lower compute setting. \n* * *\n#### Subscribe to Deep (Learning) Focus\nBy Cameron R. Wolfe · Hundreds of paid subscribers\nI contextualize and explain important topics in AI research.\nSubscribe\nBy subscribing, you agree Substack's [Terms of Use](https://substack.com/tos), and acknowledge its [Information Collection Notice](https://substack.com/ccpa#personal-data-collected) and [Privacy Policy](https://substack.com/privacy).\n[![Jakub Halmeš's avatar](https://substackcdn.com/image/fetch/$s_!Ytxr!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbc47f4-5e69-4d42-95d4-4b093703e56b_4000x4000.jpeg)](https://substack.com/profile/124459162-jakub-halmes)[![Saurabh Sinha's avatar](https://substackcdn.com/image/fetch/$s_!teLC!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F586128e3-dd1f-4b3e-9a65-bf7cbc4c37c3_144x144.png)](https://substack.com/profile/5969044-saurabh-sinha)[![Repeatuntilldeath's avatar](https://substackcdn.com/image/fetch/$s_!WwgC!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F167ed655-d63c-41b1-9ed5-870a3bbf7f2a_391x391.png)](https://substack.com/profile/20125078-repeatuntilldeath)[![Peter Xing's avatar](https://substackcdn.com/image/fetch/$s_!yOzT!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9f30792-c26a-4b49-9bd4-015e1047ecbc_533x533.jpeg)](https://substack.com/profile/17521482-peter-xing)[![Michael's avatar](https://substackcdn.com/image/fetch/$s_!AQMU!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F8a2eb040-e118-40b5-a79c-f01ad5503f4c_534x800.jpeg)](https://substack.com/profile/59416223-michael)\n155 Likes∙\n[17 Restacks](https://substack.com/note/p-152758713/restacks?utm_source=substack&utm_content=facepile-restacks)\n155\n10\n17\nShare\n#### Discussion about this post\nCommentsRestacks\n![User's avatar](https://substackcdn.com/image/fetch/$s_!TnFC!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack.com%2Fimg%2Favatars%2Fdefault-light.png)\n[![Yousef Kotp's avatar](https://substackcdn.com/image/fetch/$s_!yALW!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71725482-2e87-4e02-8a55-07a55e62e8e0_800x800.jpeg)](https://substack.com/profile/139437526-yousef-kotp?utm_source=comment)\n[Yousef Kotp](https://substack.com/profile/139437526-yousef-kotp?utm_source=substack-feed-item)\n[Feb 15, 2025](https://cameronrwolfe.substack.com/p/llm-scaling-laws/comment/93714490 \"Feb 15, 2025, 4:43 PM\")\nLiked by Cameron R. Wolfe, Ph.D.\nPerfect article, thank you so much for your writing and amazing insights!\n[Like (1)](javascript:void\\(0\\))ReplyShare\n[1 reply by Cameron R. Wolfe, Ph.D.](https://cameronrwolfe.substack.com/p/llm-scaling-laws/comment/93714490)\n[![Srivatsa Srinivas's avatar](https://substackcdn.com/image/fetch/$s_!P0iL!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab9d386d-425c-497e-a5a7-27ce13e91501_96x96.jpeg)](https://substack.com/profile/245194252-srivatsa-srinivas?utm_source=comment)\n[Srivatsa Srinivas](https://substack.com/profile/245194252-srivatsa-srinivas?utm_source=substack-feed-item)\n[Jan 19, 2025](https://cameronrwolfe.substack.com/p/llm-scaling-laws/comment/87203636 \"Jan 19, 2025, 10:43 PM\")\nLiked by Cameron R. Wolfe, Ph.D.\nHi Cameron, \nThanks for the awesome article\nYou might be interested in this <https://arxiv.org/abs/2501.04682> article for the o1/o3 section\n[Like (1)](javascript:void\\(0\\))ReplyShare\n[1 reply by Cameron R. Wolfe, Ph.D.](https://cameronrwolfe.substack.com/p/llm-scaling-laws/comment/87203636)\n[8 more comments...](https://cameronrwolfe.substack.com/p/llm-scaling-laws/comments)\nTopLatestDiscussions\n[Decoder-Only Transformers: The Workhorse of Generative LLMs](https://cameronrwolfe.substack.com/p/decoder-only-transformers-the-workhorse)\n[Building the world's most influential neural network architecture from scratch...](https://cameronrwolfe.substack.com/p/decoder-only-transformers-the-workhorse)\nMar 4, 2024 • [Cameron R. Wolfe, Ph.D.](https://substack.com/@cwolferesearch)\n181\n16\n10\n![](https://substackcdn.com/image/fetch/$s_!-1vf!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e3c9db5-400a-49de-a235-e09bc3aa3689_2392x1342.png)\n[Agent Evaluation: A Detailed Guide](https://cameronrwolfe.substack.com/p/agent-evals)\n[Best practices and common patterns for effectively evaluating AI agents...](https://cameronrwolfe.substack.com/p/agent-evals)\nMay 18 • [Cameron R. Wolfe, Ph.D.](https://substack.com/@cwolferesearch)\n338\n26\n53\n![](https://substackcdn.com/image/fetch/$s_!34pV!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43315619-4430-4543-9aea-c8c1655549c2_1928x1080.png)\n[Demystifying Reasoning Models](https://cameronrwolfe.substack.com/p/demystifying-reasoning-models)\n[Understanding reasoning models and their relation to standard LLMs...](https://cameronrwolfe.substack.com/p/demystifying-reasoning-models)\nFeb 18, 2025 • [Cameron R. Wolfe, Ph.D.](https://substack.com/@cwolferesearch)\n300\n10\n30\n![](https://substackcdn.com/image/fetch/$s_!mk9r!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F23d9c87e-b238-4fdd-996e-4ed4465b9931_2334x1282.png)\nSee all\n### Ready for more?\nSubscribe\n© 2026 Cameron R. Wolfe · [Privacy](https://substack.com/privacy) ∙ [Terms](https://substack.com/tos) ∙ [Collection notice](https://substack.com/ccpa#personal-data-collected)\n[ Start your Substack](https://substack.com/signup?utm_source=substack&utm_medium=web&utm_content=footer)[Get the app](https://substack.com/app/app-store-redirect?utm_campaign=app-marketing&utm_content=web-footer-button)\n[Substack](https://substack.com) is the home for great culture\n",
  "body_base_path": "attachments/514a31d31a2b/pages/",
  "body_markdown_path": "attachments/514a31d31a2b/pages/article.md",
  "local_html": "attachments/514a31d31a2b/pages/article.html",
  "body_source": "prior_capture",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2001.08361",
      "bytes": 2493004,
      "reused": true,
      "local_path": "papers/repair/9d7d7113c60c89afd004.pdf"
    },
    {
      "status": "ok",
      "url": "https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf",
      "bytes": 541036,
      "reused": true,
      "local_path": "papers/repair/a3e86ad05f37e40d21b6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2203.03466",
      "bytes": 3195677,
      "reused": false,
      "content_type": "application/pdf",
      "local_path": "papers/repair/61699f46540df72b1235.pdf"
    },
    {
      "status": "ok",
      "url": "https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf",
      "bytes": 582775,
      "reused": true,
      "local_path": "papers/repair/3c62359064baa28a7743.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2005.14165",
      "bytes": 6768044,
      "reused": true,
      "local_path": "papers/repair/0c1629d8d07d676c2aae.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2303.08774",
      "bytes": 5245564,
      "reused": true,
      "local_path": "papers/repair/e00f74df20cd8064349a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2203.15556",
      "bytes": 6004349,
      "reused": true,
      "local_path": "papers/repair/70766cef482201671d90.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2405.04434",
      "bytes": 1586880,
      "reused": true,
      "local_path": "papers/repair/5175a77604cd16ff9e0e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2404.19737",
      "bytes": 1725307,
      "reused": false,
      "content_type": "application/pdf",
      "local_path": "papers/repair/8d52018e3029fbd1308d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2205.11916",
      "bytes": 762445,
      "reused": true,
      "local_path": "papers/repair/7ee8d7eb1358c6773ecd.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2311.12022",
      "bytes": 617128,
      "reused": true,
      "local_path": "papers/repair/9aae134cc3cb0f937388.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1409.3215",
      "bytes": 112084,
      "reused": true,
      "local_path": "papers/repair/61d552395d0f0789263b.pdf"
    }
  ],
  "body_chars": 58025,
  "expected_body_images": 45,
  "raw_html": "attachments/514a31d31a2b/pages/source.html"
}