{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Sebastian Raschka / Ahead of AI",
  "title_listed": "Understanding Large Language Models",
  "keywords": "LLM,原理,入门",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://magazine.sebastianraschka.com/p/understanding-large-language-models",
  "publish_date": "2023-04",
  "corpus_stage": "预训练/SFT",
  "notes": "LLM原理经典入门",
  "id": "d2de6749892a",
  "collected_at": "2026-10-01T20:32:36",
  "title": "Understanding Large Language Models",
  "body_markdown": "Large language models have taken the public attention by storm – no pun intended. In just half a decade large language models – transformers – have almost completely changed the field of natural language processing. Moreover, they have also begun to revolutionize fields such as computer vision and computational biology.\nSince transformers have such a big impact on everyone’s research agenda, I wanted to flesh out a short reading list (an extended version of\nmy comment yesterday\n) for machine learning researchers and practitioners getting started.\nThe following list below is meant to be read mostly chronologically, and I am entirely focusing on academic research papers. Of course, there are many additional resources out there that are useful. For example,\nthe\nIllustrated Transformer\nby Jay Alammar;\na\nmore technical blog article\nby Lilian Weng;\na catalog and family tree\nof all major transformers to date by Xavier Amatriain;\na minimal code implementation\nof a generative language model for educational purposes by Andrej Karpathy;\na lecture series\nand\nbook chapter\nby yours truly.\nUnderstanding the Main Architecture and Tasks\nIf you are new to transformers / large language models, it makes the most sense to start at the beginning.\n(1)\nNeural Machine Translation by Jointly Learning to Align and Translate\n(2014)\nby Bahdanau, Cho, and Bengio,\nhttps://arxiv.org/abs/1409.0473\nI recommend beginning with the above paper if you have a few minutes to spare. It introduces an attention mechanism for recurrent neural networks (RNN) to improve long-range sequence modeling capabilities. This allows RNNs to translate longer sentences more accurately – the motivation behind developing the original transformer architecture later.\nSource:\nhttps://arxiv.org/abs/1409.0473\n(2)\nAttention Is All You Need\n(2017)\nby Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin,\nhttps://arxiv.org/abs/1706.03762\nThe paper above introduces the original transformer architecture consisting of an encoder- and decoder part that will become relevant as separate modules later. Moreover, this paper introduces concepts such as the scaled dot product attention mechanism, multi-head attention blocks, and positional input encoding that remain the foundation of modern transformers.\nSource:\nhttps://arxiv.org/abs/1706.03762\n(3)\nOn Layer Normalization in the Transformer Architecture\n(2020)\nby Xiong, Yang, He, K Zheng, S Zheng, Xing, Zhang, Lan, Wang, and Liu,\nhttps://arxiv.org/abs/2002.04745\nWhile the original transformer figure above (from Attention Is All You Need,\nhttps://arxiv.org/abs/1706.03762\n) is a helpful summary of the original encoder-decoder architecture, the location of the LayerNorm in this figure remains a hotly debated subject.\nFor instance, the Attention Is All You Need transformer figure places the layer normalization between the residual blocks, which doesn't match the\nofficial (updated) code implementation\naccompanying the original transformer paper. The variant shown in the Attention Is All You Need figure is known as Post-LN Transformer, and the updated code implementation defaults to the Pre-LN variant.\nThe\nLayer Normalization in the Transformer Architecture\npaper suggests that Pre-LN works better, addressing gradient problems, as shown below. Many architectures adopted this in practice, but it can result in representation collapse.\nSo, while there's still an ongoing discussion regarding using Post-LN or Pre-LN, there's also a new paper that proposes taking advantage of both worlds:\nResiDual: Transformer with Dual Residual Connections\n(\nhttps://arxiv.org/abs/2304.14802\n); whether it will turn out useful in practice remains to be seen.\nSources:\nhttps://arxiv.org/abs/1706.03762\n(left and center) and\nhttps://arxiv.org/abs/2002.04745\n(right)\n(4)\nLearning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Neural Networks\n(1991)\nby Schmidhuber,\nhttps://www.semanticscholar.org/paper/Learning-to-Control-Fast-Weight-Memories%3A-An-to-Schmidhuber/bc22e87a26d020215afe91c751e5bdaddd8e4922\nThis paper is recommended for those interested in historical tidbits and early approaches fundamentally similar to modern transformers.\nFor instance, in 1991, which is about two-and-a-half decades before the original transformer paper above (\"Attention Is All You Need\"), Juergen Schmidhuber proposed an alternative to recurrent neural networks called\nFast Weight Programmers\n(FWP). The FWP approach involves a feedforward neural network that slowly learns by gradient descent to program the changes of the fast weights of another neural network.\nThe analogy to modern transformers is explained\nin this blog post\nas follows:\nIn today's Transformer terminology, FROM and TO are called key and value, respectively. The INPUT to which the fast net is applied is called the query. Essentially, the query is processed by the fast weight matrix, which is a sum of outer products of keys and values (ignoring normalizations and projections). Since all operations of both networks are differentiable, we obtain end-to-end differentiable active control of fast weight changes through additive outer products or second order tensor products.[FWP0-3a] Hence the slow net can learn by gradient descent to rapidly modify the fast net during sequence processing. This is mathematically equivalent (apart from normalization) to what was later called Transformers with linearized self-attention (or linear Transformers).\nAs mentioned in the blog post excerpt above, this approach is now called \"linear Transformers\" or \"Transformers with linearized self-attention\" via the more recent papers\nTransformers are RNNs: Fast Autoregressive Transformers with Linear Attention\nand\nRethinking Attention with Performers\nthat appeared on arXiv in 2020.\nIn 2021, the\nLinear Transformers Are Secretly Fast Weight Programmers\npaper then explicitly showed the equivalence between linearized self-attention and the fast weight programmers from the 1990s.\nSource: Annotated figure based on\nhttps://people.idsia.ch//~juergen/fast-weight-programmer-1991-transformer.html#sec2\n(5)\nUniversal Language Model Fine-tuning for Text Classification\n(2018)\nby Howard and Ruder,\nhttps://arxiv.org/abs/1801.06146\nThis is another paper that's very interesting from a historical perspective. While it was written one year after the original\nAttention Is All You Need\ntransformer was released, it doesn't involve transformers but instead focuses on recurrent neural networks. However, it's still noteworthy since it effectively proposed pretraining language models and transfer learning for downstream tasks.\nWhile transfer learning was already established in computer vision, it wasn't yet prevalent in natural language processing (NLP). ULMFit was among the first papers to demonstrate that pretraining a language model and finetuning it on a specific task could yield state-of-the-art results in many NLP tasks.\nThe three-stage process for finetuning the language models suggested by ULMFit was as follows:\nTrain a language model on a large corpus of text.\nFinetune this pretrained language model on task-specific data, allowing it to adapt to the specific style and vocabulary of the text.\nFinetune a classifier on the task-specific data with gradual unfreezing of layers to avoid catastrophic forgetting.\nThis recipe -- training a language model on a large corpus and then finetuning it on a downstream task -- is the central approach used in transformer-based models and foundation models like BERT, GPT-2/3/4, RoBERTa, and others.\nHowever, the gradual unfreezing, a key part of ULMFiT, is usually not routinely done in practice when working with transformer architectures, where all layers are typically finetuned at once.\nSource:\nhttps://arxiv.org/abs/1801.06146\n(6)\nBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding\n(2018)\nby Devlin, Chang, Lee, and Toutanova,\nhttps://arxiv.org/abs/1810.04805\nFollowing the original transformer architecture, large language model research started to bifurcate in two directions: encoder-style transformers for predictive modeling tasks such as text classification and decoder-style transformers for generative modeling tasks such as translation, summarization, and other forms of text creation.\nThe BERT paper above introduces the original concept of masked-language modeling, and next-sentence prediction. It still is the most influential encoder-style architecture. If you are interested in this research branch, I recommend following up with\nRoBERTa\n, which simplified the pretraining objectives by removing the next-sentence prediction tasks.\nSource:\nhttps://arxiv.org/abs/1810.04805\n(7)\nImproving Language Understanding by Generative Pre-Training\n(2018) by Radford and Narasimhan\n,\nhttps://www.semanticscholar.org/paper/Improving-Language-Understanding-by-Generative-Radford-Narasimhan/cd18800a0fe0b668a1cc19f2ec95b5003d0a5035\nThe original GPT paper introduced the popular decoder-style architecture and pretraining via next-word prediction. Where BERT can be considered a bidirectional transformer due to its masked language model pretraining objective, GPT is a unidirectional, autoregressive model. While GPT embeddings can also be used for classification, the GPT approach is at the core of today’s most influential LLMs, such as chatGPT.\nIf you are interested in this research branch, I recommend following up with the\nGPT-2\nand\nGPT-3\npapers. These two papers illustrate that LLMs are capable of zero- and few-shot learning and highlight the emergent abilities of LLMs. GPT-3 is also still a popular baseline and base model for training current-generation LLMs such as ChatGPT – we will cover the InstructGPT approach that lead to ChatGPT later as a separate entry.\nSource:\nhttps://www.semanticscholar.org/paper/Improving-Language-Understanding-by-Generative-Radford-Narasimhan/cd18800a0fe0b668a1cc19f2ec95b5003d0a5035\n(8)\nBART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension\n(2019)\nby Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy, Stoyanov, and Zettlemoyer,\nhttps://arxiv.org/abs/1910.13461\n.\nAs mentioned earlier, BERT-type encoder-style LLMs are usually preferred for predictive modeling tasks, whereas GPT-type decoder-style LLMs are better at generating texts. To get the best of both worlds, the BART paper above combines both the encoder and decoder parts (not unlike the original transformer – the second paper in this list).\nSource:\nhttps://arxiv.org/abs/1910.13461\n(9)\nHarnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond\n(2023)\nby Yang, Jin, Tang, Han, Feng, Jiang, Yin, and Hu,\nhttps://arxiv.org/abs/2304.13712\nThis is not a research paper but probably the best general architecture survey to date illustrating how the different architectures evolved. However, next to discussing BERT-style masked language models (encoders) and GPT-style autoregressive language models (decoders), it also provides useful discussions and guidance regarding pretraining and finetuning data.\nThe evolutionary tree of modern LLMs via\nhttps://arxiv.org/abs/2304.13712\n.\nScaling Laws and Improving Efficiency\nIf you want to learn more about the various techniques to improve the efficiency of transformers, I recommend the\n2020\nEfficient Transformers: A Survey\npaper followed by the\n2023\nA Survey on Efficient Training of Transformers\npaper.\nIn addition, below are papers that I found particularly interesting and worth reading.\n(10)\nFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\n(2022), by Dao, Fu, Ermon, Rudra, and Ré,\nhttps://arxiv.org/abs/2205.14135\n.\nWhile most transformer papers don’t bother about replacing the original scaled dot product mechanism for implementing self-attention, FlashAttention is one mechanism I have seen most often referenced lately.\nSource:\nhttps://arxiv.org/abs/2205.14135\n(11)\nCramming: Training a Language Model on a Single GPU in One Day (2022)\nby Geiping and Goldstein,\nhttps://arxiv.org/abs/2212.14034\n.\nIn this paper, the researchers trained a masked language model / encoder-style LLM (here: BERT) for 24h on a single GPU. For comparison, the original 2018 BERT paper trained it on 16 TPUs for four days. An interesting insight is that while smaller models have higher throughput, smaller models also learn less efficiently. Thus, larger models do not require more training time to reach a specific predictive performance threshold.\nSource:\nhttps://arxiv.org/abs/2212.14034\n(12)\nLoRA: Low-Rank Adaptation of Large Language Models (2021)\nby Hu, Shen, Wallis, Allen-Zhu, Li, L Wang, S Wang, and Chen,\nhttps://arxiv.org/abs/2106.09685\n.\nModern large language models that are pretrained on large datasets show emergent abilities and perform well on various tasks, including language translation, summarization, coding, and Q&A. However, if we want to improve the ability of transformers on domain-specific data and specialized tasks, it’s worthwhile to finetune transformers.\nLow-rank adaptation (LoRA) is one of the most influential approaches for finetuning large language models in a parameter-efficient manner. While other methods for parameter-efficient finetuning exist (see the survey below), LoRA is particularly worth highlighting as it is both an elegant and very general method that can be applied to other types of models as well.\nWhile the weights of a pretrained model have full rank on the pretrained tasks, the LoRA authors point out that pretrained large language models have a low “intrinsic dimension” when they are adapted to a new task. So, the main idea behind LoRA is to decompose the weight changes, ΔW, into a lower-rank representation, which is more parameter efficient.\nAn illustration of LoRA alongside its performance from\nhttps://arxiv.org/abs/2106.09685\n.\n(13)\nScaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning (2022)\nby Lialin, Deshpande, and Rumshisky,\nhttps://arxiv.org/abs/2303.15647\n.\nModern large language models that are pretrained on large datasets show emergent abilities and perform well on various tasks, including language translation, summarization, coding, and Q&A. However, if we want to improve the ability of transformers on domain-specific data and specialized tasks, it’s worthwhile to finetune transformers. This survey reviews more than 40 papers on parameter-efficient finetuning methods (including popular techniques such as prefix tuning, adapters, and low-rank adaptation) to make finetuning (very) computationally efficient.\nSource:\nhttps://arxiv.org/abs/2303.15647\n(14)\nScaling Language Models: Methods, Analysis & Insights from Training Gopher\n(2022)\nby Rae and colleagues (78 co-authors!),\nhttps://arxiv.org/abs/2112.11446\nGopher\nis a particularly nice paper including tons of analysis to understand LLM training. Here, the researchers trained a 280 billion parameter model with 80 layers on 300 billions tokens. This includes interesting architecture modifications such as using RMSNorm (Root Mean Square Normalization) instead of LayerNorm (Layer Normalization). Both LayerNorm and RMSNorm are preferred over BatchNorm since they don't depend on the batch size and doesn't require synchronization, which is an advantage in distributed settings with smaller batch sizes. However, RMSNorm is generally said to stabilize the training in deeper architectures.\nBesides interesting tidbits such the ones above, the main focus of this paper is the analysis of task performance for different scales. The evaluation on 152 diverse tasks reveal that increasing model sizes benefits tasks like comprehension, fact-checking, and the identification of toxic language the most. However, tasks related to logical and mathematical reasoning benefit less from architecture scaling.\nSource: Figure from\nhttps://arxiv.org/abs/2112.11446\n(15)\nTraining Compute-Optimal Large Language Models\n(2022)\nby Hoffmann, Borgeaud, Mensch, Buchatskaya, Cai, Rutherford, de Las Casas, Hendricks, Welbl, Clark, Hennigan, Noland, Millican, van den Driessche, Damoc, Guy, Osindero, Simonyan, Elsen, Rae, Vinyals, and Sifre,\nhttps://arxiv.org/abs/2203.15556\n.\nThis paper introduces the 70-billion parameter Chinchilla model that outperforms the popular 175-billion parameter GPT-3 model on generative modeling tasks. However, its main punchline is that contemporary large language models are “significantly undertrained.”\nThe paper defines the linear scaling law for large language model training. For example, while Chinchilla is only half the size of GPT-3, it outperformed GPT-3 because it was trained on 1.4 trillion (instead of just 300 billion) tokens. In other words, the number of training tokens is as vital as the model size.\nSource:\nhttps://arxiv.org/abs/2203.15556\n(16)\nPythia\n:\nA Suite for Analyzing Large Language Models Across Training and Scaling\n(2023)\nby Biderman, Schoelkopf, Anthony, Bradley, O'Brien, Hallahan, Khan, Purohit, Prashanth, Raff, Skowron, Sutawika, and van der Wal,\nhttps://arxiv.org/abs/2304.01373\nPythia is a suite of open-source LLMs (70M to 12B parameters) to study how LLMs evolve over the course of training.\nThe architecture is similar to GPT-3 but includes some improvements, for example, Flash Attention (like\nLLaMA\n) and Rotary Positional Embeddings (like\nPaLM\n). Pythia was trained on\nThe Pile dataset\n(825 Gb) for 300 B tokens (~1 epoch on regular PILE, ~1.5 epochs on deduplicated PILE.\nThe Pythia model suite via\nhttps://arxiv.org/abs/2304.01373\n.\nThe main insights of the Pythia study are as follows:\nTraining on duplicated data (due to how LLMs are trained, this means training for more than one epoch) does not benefit or hurt performance.\nTraining order does not influence memorization. This is unfortunate because if the opposite were true, we could mitigate undesirable verbatim memorization issues by reordering the training data.\nPretrained term frequency influences task performance. For instance, few-shot accuracy tends to be higher for more frequent terms.\nDoubling the batch size halves the training time but doesn't hurt convergence.\nAlignment – Steering Large Language Models to Intended Goals and Interests\nIn recent years, we have seen many relatively capable large language models that can generate realistic texts (for example, GPT-3 and Chinchilla, among others). It seems that we have reached a ceiling in terms of what we can achieve with the commonly used pretraining paradigms.\nTo make language models more helpful and reduce misinformation and harmful language, researchers designed additional training paradigms to fine-tune the pretrained base models.\n(17)\nTraining Language Models to Follow Instructions with Human Feedback\n(2022)\nby Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray, Schulman, Hilton, Kelton, Miller, Simens, Askell, Welinder, Christiano, Leike, and Lowe,\nhttps://arxiv.org/abs/2203.02155\n.\nIn this so-called InstructGPT paper, the researchers use a reinforcement learning mechanism with humans in the loop (RLHF). They start with a pretrained GPT-3 base model and fine-tune it further using supervised learning on prompt-response pairs generated by humans (Step 1). Next, they ask humans to rank model outputs to train a reward model (step 2). Finally, they use the reward model to update the pretrained and fine-tuned GPT-3 model using reinforcement learning via proximal policy optimization (step 3).\nAs a sidenote, this paper is also known as the paper describing the idea behind ChatGPT – according to the recent rumors, ChatGPT is a scaled-up version of InstructGPT that has been fine-tuned on a larger dataset.\nSource:\nhttps://arxiv.org/abs/2203.02155\n(18)\nConstitutional AI: Harmlessness from AI Feedback\n(2022\n) by Yuntao, Saurav, Sandipan, Amanda, Jackson, Jones, Chen, Anna, Mirhoseini, McKinnon, Chen, Olsson, Olah, Hernandez, Drain, Ganguli, Li, Tran-Johnson, Perez, Kerr, Mueller, Ladish, Landau, Ndousse, Lukosuite, Lovitt, Sellitto, Elhage, Schiefer, Mercado, DasSarma, Lasenby, Larson, Ringer, Johnston, Kravec, El Showk, Fort, Lanham, Telleen-Lawton, Conerly, Henighan, Hume, Bowman, Hatfield-Dodds, Mann, Amodei, Joseph, McCandlish, Brown, Kaplan,\nhttps://arxiv.org/abs/2212.08073\n.\nIn this paper, the researchers are taking the alignment idea one step further, proposing a training mechanism for creating a “harmless” AI system. Instead of direct human supervision, the researchers propose a self-training mechanism that is based on a list of rules (which are provided by a human). Similar to the InstructGPT paper mentioned above, the proposed method uses a reinforcement learning approach.\nSource:\nhttps://arxiv.org/abs/2212.08073\n(19)\nSelf-Instruct: Aligning Language Model with Self Generated Instruction\n(2022)\nby Wang, Kordi, Mishra, Liu, Smith, Khashabi, and Hajishirzi,\nhttps://arxiv.org/abs/2212.10560\nInstruction finetuning is how we get from GPT-3-like pretrained base models to more capable LLMs like ChatGPT. And open-source human-generated instruction datasets like\ndatabricks-dolly-15k\ncan help make this possible. But how do we scale this? One way is bootstrapping an LLM off its own generations.\nSelf-Instruct is one (almost annotation-free) way to align pretrained LLMs with instructions.\nHow does this work? In a nutshell, it's a 4-step process:\nSeed task pool with a set of human-written instructions (175 in this case) and sample instructions.\nUse a pretrained LLM (like GPT-3) to determine the task category.\nGiven the new instruction, let a pretrained LLM generate the response.\nCollect, prune, and filter the responses before adding them to the task pool.\nAnnotated version of the self-instruct method from\nhttps://arxiv.org/abs/2212.10560\n.\nIn practice, this works relatively well based on the ROUGE scores.\nFor example, a Self-Instruct-finetuned LLM outperforms the GPT-3 base LLM (1) and can compete with an LLM pretrained on a large human-written instruction set (2). And self-instruct can also benefit LLMs that were already finetuned on human instructions (3).\nBut of course, the gold standard for evaluating LLMs is to ask human raters. Based on human evaluation, Self-Instruct outperforms base LLM, and LLMs trained on human instruction datasets in supervised fashion (SuperNI, T0 Trainer). But interestingly, Self-Instruct does not outperform methods trained via reinforcement learning with human feedback (RLHF)\nWhich is more promising, human-generated instruction datasets or self-instruct-datasets? I vote for both. Why not start with a human-generated instruction dataset like the 15k instructions from\ndatabricks-dolly-15k\nand then scale this with self-instruct?\nReinforcement Learning with Human Feedback (RLHF)\nFor additional explanations of Reinforcement Learning with Human Feedback (RLHF), plus papers on proximal policy optimization for implementing RLHF, please see my more detailed article below:\nLLM Training: RLHF and Its Alternatives\nSebastian Raschka, PhD\n·\nSeptember 10, 2023\nI frequently reference a process called Reinforcement Learning with Human Feedback (RLHF) when discussing LLMs, whether in the research news or tutorials. RLHF is an integral part of the modern LLM training pipeline due to its ability to incorporate human preferences into the optimization landscape, which can improve the model's helpfulness and safety.\nRead full story\nConclusion and Further Reading\nI tried to keep the list above nice and concise, focusing on the top-10 papers (plus 3 bonus papers on RLHF) to understand the design, constraints, and evolution behind contemporary large language models.\nFor further reading, I suggest following the references in the papers mentioned above. Or, to give you some additional pointers, here are some additional resources (these lists are not comprehensive):\nOpen-source alternatives to GPT\nBLOOM: A 176B-Parameter Open-Access Multilingual Language Model\n(2022),\nhttps://arxiv.org/abs/2211.05100\nOPT: Open Pre-trained Transformer Language Models\n(2022),\nhttps://arxiv.org/abs/2205.01068\nUL2:\nUnifying Language Learning Paradigms\n(2022)\n,\nhttps://arxiv.org/abs/2205.05131\nChatGPT alternatives\nLaMDA: Language Models for Dialog Applications\n(2022),\nhttps://arxiv.org/abs/2201.08239\n(Bloomz)\nCrosslingual Generalization through Multitask Finetuning\n(2022),\nhttps://arxiv.org/abs/2211.01786\n(Sparrow)\nImproving Alignment of Dialogue Agents via Targeted Human Judgements\n(2022),\nhttps://arxiv.org/abs/2209.14375\nBlenderBot 3: A Deployed Conversational Agent that Continually Learns to Responsibly Engage\n,\nhttps://arxiv.org/abs/2208.03188\nLarge language models in computational biology\nProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing\n(2021),\nhttps://arxiv.org/abs/2007.06225\nHighly Accurate Protein Structure Prediction with AlphaFold\n(2021),\nhttps://www.nature.com/articles/s41586-021-03819-2\nLarge Language Models Generate Functional Protein Sequences Across Diverse Families\n(2023),\nhttps://www.nature.com/articles/s41587-022-01618-2\nThis magazine is a personal passion project. For those who wish to support me, please consider purchasing a copy of my\nBuild a Large Language Model (From Scratch) book\n. (I am confident that you'll get lots out of this book as it explains how LLMs work in a level of detail that is not found anywhere else.)\nBuild a Large Language Model (From Scratch)\nnow available on Amazon\nIf you read the book and have a few minutes to spare, I'd really appreciate a\nbrief review\n. It helps us authors a lot!\nAlternatively, I also recently enabled the paid subscription option on Substack to support this magazine directly.\nSubscribe",
  "metadata": {
    "title": "Understanding Large Language Models",
    "description": "Explore the transformative power of large language models in AI. Dive into a curated reading list for ML enthusiasts. Discover the impact of transformers on NLP, vision, and biology.",
    "keywords": null,
    "author": "Sebastian Raschka, PhD",
    "og:type": "article",
    "og:title": "Understanding Large Language Models",
    "og:description": "A Cross-Section of the Most Relevant Literature To Get Up to Speed",
    "og:image": "https://substackcdn.com/image/fetch/$s_!RaNK!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9a0766d-2e52-4af0-96c5-3e07a30d6ecb_1868x1130.png",
    "og:url": "https://magazine.sebastianraschka.com/p/understanding-large-language-models",
    "twitter:title": "Understanding Large Language Models",
    "twitter:description": "A Cross-Section of the Most Relevant Literature To Get Up to Speed",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!Bz-Y!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fsebastianraschka.substack.com%2Fapi%2Fv1%2Fpost_preview%2F115060492%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2024-12-22T13:33:31.307Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Ss64!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff907c364-b179-4112-a2b9-50c389d0377b_904x396.png",
      "bytes": 29799,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-7d1145935f36a177d67c.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Ss64!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff907c364-b179-4112-a2b9-50c389d0377b_904x396.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!GCTD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc820b36a-8e07-4eea-8975-d7e391006d52_824x690.png",
      "bytes": 49959,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-90ef688600b01d84237c.png",
      "src": "https://substackcdn.com/image/fetch/$s_!GCTD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc820b36a-8e07-4eea-8975-d7e391006d52_824x690.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!JOyz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a0a1e39-de22-4a95-8a42-b4040cf33b54_2040x956.png",
      "bytes": 117380,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-39744abc1c8509cd9967.png",
      "src": "https://substackcdn.com/image/fetch/$s_!JOyz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a0a1e39-de22-4a95-8a42-b4040cf33b54_2040x956.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!xKpc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2217256f-c000-49bf-878e-7a3e062c7fb8_884x894.png",
      "bytes": 65325,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-08151e7c09f13244e828.png",
      "src": "https://substackcdn.com/image/fetch/$s_!xKpc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2217256f-c000-49bf-878e-7a3e062c7fb8_884x894.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!G1sW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2d217b3-4596-42bb-a070-c7c1ae9b90cb_1762x850.png",
      "bytes": 151707,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-04a238d05648f74b7240.png",
      "src": "https://substackcdn.com/image/fetch/$s_!G1sW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2d217b3-4596-42bb-a070-c7c1ae9b90cb_1762x850.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!f1ZR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89243595-9867-4420-b753-802ca1226518_1368x830.png",
      "bytes": 200006,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-a7eff6cf3b7658093b4f.png",
      "src": "https://substackcdn.com/image/fetch/$s_!f1ZR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89243595-9867-4420-b753-802ca1226518_1368x830.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!--fa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F573a9c49-2ff3-4a14-98b6-6b8a44333601_1314x658.png",
      "bytes": 100773,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-ea57212c662da4a4d4a7.png",
      "src": "https://substackcdn.com/image/fetch/$s_!--fa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F573a9c49-2ff3-4a14-98b6-6b8a44333601_1314x658.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!cnoO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7797ec59-f683-4f30-8281-ed0503076b6b_1004x432.png",
      "bytes": 56754,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-120518d1a6323d3b98b4.png",
      "src": "https://substackcdn.com/image/fetch/$s_!cnoO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7797ec59-f683-4f30-8281-ed0503076b6b_1004x432.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!w0MI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1f675dc-d643-405d-8111-eab067b7012f_1538x1188.png",
      "bytes": 159150,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-1d2ab446c9533910a5a1.png",
      "src": "https://substackcdn.com/image/fetch/$s_!w0MI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1f675dc-d643-405d-8111-eab067b7012f_1538x1188.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!92eZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659f5e23-5641-406f-9a08-a2461ffff345_1068x614.png",
      "bytes": 95180,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-269d524a1dcfdc7e77fe.png",
      "src": "https://substackcdn.com/image/fetch/$s_!92eZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659f5e23-5641-406f-9a08-a2461ffff345_1068x614.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!ElDh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8197f14a-dcb6-471e-8958-09032c75e73f_1310x620.png",
      "bytes": 129881,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-3e600fdb807b328fbdbc.png",
      "src": "https://substackcdn.com/image/fetch/$s_!ElDh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8197f14a-dcb6-471e-8958-09032c75e73f_1310x620.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!-mrt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F542db321-11fe-4641-8f63-2fa7ac7a6536_2302x726.png",
      "bytes": 96673,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-ce364051ee08f2f54650.png",
      "src": "https://substackcdn.com/image/fetch/$s_!-mrt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F542db321-11fe-4641-8f63-2fa7ac7a6536_2302x726.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!O4Fa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c015d7-1345-48b4-8170-fd9bb636b92f_2102x1078.png",
      "bytes": 76223,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-f232bf7f59440f23fbad.png",
      "src": "https://substackcdn.com/image/fetch/$s_!O4Fa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c015d7-1345-48b4-8170-fd9bb636b92f_2102x1078.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!BVt9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86050fd8-33e1-483e-bdb8-daa1873fd018_1788x1408.png",
      "bytes": 147651,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-41f299906c9b231ea828.png",
      "src": "https://substackcdn.com/image/fetch/$s_!BVt9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86050fd8-33e1-483e-bdb8-daa1873fd018_1788x1408.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!biVl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760d60f7-c4e6-4a8b-a540-09c8d1ffef36_1196x744.png",
      "bytes": 94910,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-3659b5edd57eb8c7c61e.png",
      "src": "https://substackcdn.com/image/fetch/$s_!biVl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760d60f7-c4e6-4a8b-a540-09c8d1ffef36_1196x744.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!L-Fu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf0be27f-4d1b-4761-9019-b04735c3bd25_1854x648.jpeg",
      "bytes": 91797,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-e5b481c933410dbde80a.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!L-Fu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf0be27f-4d1b-4761-9019-b04735c3bd25_1854x648.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!bc_C!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79d09d26-5644-4830-a1de-a94c7149d50c_1294x822.png",
      "bytes": 147754,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-fd63f2a4ce35f8016439.png",
      "src": "https://substackcdn.com/image/fetch/$s_!bc_C!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79d09d26-5644-4830-a1de-a94c7149d50c_1294x822.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!h1rb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdebca6d8-b504-4645-ac80-8d9aaabccd3b_1466x628.png",
      "bytes": 71676,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-770948737285b724d5c6.png",
      "src": "https://substackcdn.com/image/fetch/$s_!h1rb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdebca6d8-b504-4645-ac80-8d9aaabccd3b_1466x628.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!t9yF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88f7c2ca-5592-423d-b551-9794e325bafc_2218x1004.png",
      "bytes": 324645,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-4197dd295b67048e13b8.png",
      "src": "https://substackcdn.com/image/fetch/$s_!t9yF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88f7c2ca-5592-423d-b551-9794e325bafc_2218x1004.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!kAkh!,w_280,h_280,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe06a33c8-cdbd-4f5e-8380-86fb71a075c8_2216x1232.png",
      "bytes": 11769,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-ebfd8414e924a91ed3c5.png",
      "src": "https://substackcdn.com/image/fetch/$s_!kAkh!,w_280,h_280,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe06a33c8-cdbd-4f5e-8380-86fb71a075c8_2216x1232.png",
      "alt": "LLM Training: RLHF and Its Alternatives",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!woQp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png",
      "bytes": 121470,
      "reused": false,
      "local_path": "attachments/d2de6749892a/images/body-1c343ecc9eadd930f4bd.png",
      "src": "https://substackcdn.com/image/fetch/$s_!woQp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.linkedin.com/feed/update/urn:li:activity:7028449312300834816?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7028449312300834816%2C7028519126105030656%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287028519126105030656%2Curn%3Ali%3Aactivity%3A7028449312300834816%29",
      "label": "my comment yesterday",
      "type": "a"
    },
    {
      "href": "http://jalammar.github.io/illustrated-transformer/",
      "label": "Illustrated Transformer",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2020-04-07-the-transformer-family/",
      "label": "more technical blog article",
      "type": "a"
    },
    {
      "href": "https://amatriain.net/blog/transformer-models-an-introduction-and-catalog-2d1e9039f376/",
      "label": "a catalog and family tree",
      "type": "a"
    },
    {
      "href": "https://github.com/karpathy/nanoGPT",
      "label": "a minimal code implementation",
      "type": "a"
    },
    {
      "href": "https://sebastianraschka.com/blog/2021/dl-course.html",
      "label": "a lecture series",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/machine-learning-book/tree/main/ch16",
      "label": "book chapter",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1409.0473",
      "label": "https://arxiv.org/abs/1409.0473",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!Ss64!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff907c364-b179-4112-a2b9-50c389d0377b_904x396.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.03762",
      "label": "https://arxiv.org/abs/1706.03762",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!GCTD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc820b36a-8e07-4eea-8975-d7e391006d52_824x690.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2002.04745",
      "label": "https://arxiv.org/abs/2002.04745",
      "type": "a"
    },
    {
      "href": "https://github.com/tensorflow/tensor2tensor/commit/f5c9b17e617ea9179b7d84d36b1e8162cb369f25",
      "label": "official (updated) code implementation",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2304.14802",
      "label": "https://arxiv.org/abs/2304.14802",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!JOyz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a0a1e39-de22-4a95-8a42-b4040cf33b54_2040x956.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.semanticscholar.org/paper/Learning-to-Control-Fast-Weight-Memories%3A-An-to-Schmidhuber/bc22e87a26d020215afe91c751e5bdaddd8e4922",
      "label": "https://www.semanticscholar.org/paper/Learning-to-Control-Fast-Weight-Memories%3A-An-to-Schmidhuber/bc22e87a26d020215afe",
      "type": "a"
    },
    {
      "href": "https://people.idsia.ch//~juergen/fast-weight-programmer-1991-transformer.html",
      "label": "in this blog post",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2006.16236",
      "label": "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.14794",
      "label": "Rethinking Attention with Performers",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2102.11174",
      "label": "Linear Transformers Are Secretly Fast Weight Programmers",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!xKpc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2217256f-c000-49bf-878e-7a3e062c7fb8_884x894.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1801.06146",
      "label": "https://arxiv.org/abs/1801.06146",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!G1sW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2d217b3-4596-42bb-a070-c7c1ae9b90cb_1762x850.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1810.04805",
      "label": "https://arxiv.org/abs/1810.04805",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1907.11692",
      "label": "RoBERTa",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!f1ZR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89243595-9867-4420-b753-802ca1226518_1368x830.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.semanticscholar.org/paper/Improving-Language-Understanding-by-Generative-Radford-Narasimhan/cd18800a0fe0b668a1cc19f2ec95b5003d0a5035",
      "label": "https://www.semanticscholar.org/paper/Improving-Language-Understanding-by-Generative-Radford-Narasimhan/cd18800a0fe0b668",
      "type": "a"
    },
    {
      "href": "https://www.semanticscholar.org/paper/Language-Models-are-Unsupervised-Multitask-Learners-Radford-Wu/9405cc0d6169988371b2755e573cc28650d14dfe",
      "label": "GPT-2",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2005.14165",
      "label": "GPT-3",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!--fa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F573a9c49-2ff3-4a14-98b6-6b8a44333601_1314x658.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1910.13461",
      "label": "https://arxiv.org/abs/1910.13461",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2304.13712",
      "label": "https://arxiv.org/abs/2304.13712",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!w0MI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1f675dc-d643-405d-8111-eab067b7012f_1538x1188.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.06732",
      "label": "2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2302.01107",
      "label": "2023",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2205.14135",
      "label": "https://arxiv.org/abs/2205.14135",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!92eZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659f5e23-5641-406f-9a08-a2461ffff345_1068x614.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2212.14034",
      "label": "https://arxiv.org/abs/2212.14034",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!ElDh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8197f14a-dcb6-471e-8958-09032c75e73f_1310x620.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2106.09685",
      "label": "https://arxiv.org/abs/2106.09685",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!-mrt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F542db321-11fe-4641-8f63-2fa7ac7a6536_2302x726.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2303.15647",
      "label": "https://arxiv.org/abs/2303.15647",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!O4Fa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c015d7-1345-48b4-8170-fd9bb636b92f_2102x1078.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2112.11446",
      "label": "https://arxiv.org/abs/2112.11446",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!BVt9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86050fd8-33e1-483e-bdb8-daa1873fd018_1788x1408.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2203.15556",
      "label": "https://arxiv.org/abs/2203.15556",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!biVl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F760d60f7-c4e6-4a8b-a540-09c8d1ffef36_1196x744.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2304.01373",
      "label": "https://arxiv.org/abs/2304.01373",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2302.13971",
      "label": "LLaMA",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2204.02311",
      "label": "PaLM",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2101.00027",
      "label": "The Pile dataset",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!L-Fu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf0be27f-4d1b-4761-9019-b04735c3bd25_1854x648.jpeg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2203.02155",
      "label": "https://arxiv.org/abs/2203.02155",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!bc_C!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79d09d26-5644-4830-a1de-a94c7149d50c_1294x822.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2212.08073",
      "label": "https://arxiv.org/abs/2212.08073",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!h1rb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdebca6d8-b504-4645-ac80-8d9aaabccd3b_1466x628.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2212.10560",
      "label": "https://arxiv.org/abs/2212.10560",
      "type": "a"
    },
    {
      "href": "https://github.com/databrickslabs/dolly/tree/master/data",
      "label": "databricks-dolly-15k",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!t9yF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88f7c2ca-5592-423d-b551-9794e325bafc_2218x1004.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://magazine.sebastianraschka.com/p/llm-training-rlhf-and-its-alternatives",
      "label": "LLM Training: RLHF and Its Alternatives",
      "type": "a"
    },
    {
      "href": "https://substack.com/profile/27393275-sebastian-raschka-phd",
      "label": "Sebastian Raschka, PhD",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2211.05100",
      "label": "https://arxiv.org/abs/2211.05100",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2205.01068",
      "label": "https://arxiv.org/abs/2205.01068",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2205.05131",
      "label": "https://arxiv.org/abs/2205.05131",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2201.08239",
      "label": "https://arxiv.org/abs/2201.08239",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2211.01786",
      "label": "https://arxiv.org/abs/2211.01786",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2209.14375",
      "label": "https://arxiv.org/abs/2209.14375",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2208.03188",
      "label": "https://arxiv.org/abs/2208.03188",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2007.06225",
      "label": "https://arxiv.org/abs/2007.06225",
      "type": "a"
    },
    {
      "href": "https://www.nature.com/articles/s41586-021-03819-2",
      "label": "https://www.nature.com/articles/s41586-021-03819-2",
      "type": "a"
    },
    {
      "href": "https://www.nature.com/articles/s41587-022-01618-2",
      "label": "https://www.nature.com/articles/s41587-022-01618-2",
      "type": "a"
    },
    {
      "href": "https://amzn.to/4fqvn0D",
      "label": "Build a Large Language Model (From Scratch) book",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!woQp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167",
      "label": "Build a Large Language Model (From Scratch)",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167",
      "label": "brief review",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:23Z",
  "source_url": "https://magazine.sebastianraschka.com/p/understanding-large-language-models",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1409.0473",
      "bytes": 444482,
      "reused": false,
      "local_path": "papers/repair/b6909588ff7b9ebf375f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.03762",
      "bytes": 2215244,
      "reused": true,
      "local_path": "papers/repair/c3ab01e47e614fe3cf33.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2002.04745",
      "bytes": 779713,
      "reused": false,
      "local_path": "papers/repair/eeee73d530f66e641d46.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2304.14802",
      "bytes": 692382,
      "reused": false,
      "local_path": "papers/repair/f9a491a027b51ea0a83a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2006.16236",
      "bytes": 1728131,
      "reused": false,
      "local_path": "papers/repair/7c35a297124eaffc847c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.14794",
      "bytes": 5880177,
      "reused": false,
      "local_path": "papers/repair/26948568ccf004004970.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2102.11174",
      "bytes": 1452993,
      "reused": false,
      "local_path": "papers/repair/d20eba8532a8d6bf2885.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1801.06146",
      "bytes": 979417,
      "reused": false,
      "local_path": "papers/repair/f99f602eebf2965d0ed0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1810.04805",
      "bytes": 775166,
      "reused": true,
      "local_path": "papers/repair/33769e0bafc9f0c73baa.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1907.11692",
      "bytes": 209675,
      "reused": false,
      "local_path": "papers/repair/8acd467c11db68744d3f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2005.14165",
      "bytes": 6768044,
      "reused": true,
      "local_path": "papers/repair/0c1629d8d07d676c2aae.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1910.13461",
      "bytes": 296072,
      "reused": false,
      "local_path": "papers/repair/006bd94b666a1c81bc24.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2304.13712",
      "bytes": 2381806,
      "reused": false,
      "local_path": "papers/repair/af8960e4b8a740db7668.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.06732",
      "bytes": 847824,
      "reused": true,
      "local_path": "papers/repair/5442c18a5d3ec7c8eca5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2302.01107",
      "bytes": 366251,
      "reused": true,
      "local_path": "papers/repair/f08b9d9967e83d2d11a7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2205.14135",
      "bytes": 2630825,
      "reused": true,
      "local_path": "papers/repair/bc0de6a7863ada2c8edb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2212.14034",
      "bytes": 1078283,
      "reused": false,
      "local_path": "papers/repair/fff9233a973a112dcb6a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2106.09685",
      "bytes": 1609513,
      "reused": true,
      "local_path": "papers/repair/e7d2fed957472d7677ec.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2303.15647",
      "bytes": 1697737,
      "reused": true,
      "local_path": "papers/repair/66f57b1fa256190f7112.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2112.11446",
      "bytes": 3412401,
      "reused": false,
      "local_path": "papers/repair/4142355b96af36246d11.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2203.15556",
      "bytes": 6004349,
      "reused": true,
      "local_path": "papers/repair/70766cef482201671d90.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2304.01373",
      "bytes": 1091147,
      "reused": true,
      "local_path": "papers/repair/3c082b260d9eb0b3488d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2302.13971",
      "bytes": 726566,
      "reused": true,
      "local_path": "papers/repair/c0170963c7095dc7febf.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2204.02311",
      "bytes": 2711295,
      "reused": false,
      "local_path": "papers/repair/b20910edbfba7f89d65e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2101.00027",
      "bytes": 1528989,
      "reused": false,
      "local_path": "papers/repair/80ab88d0830ceb7017b0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2203.02155",
      "bytes": 1797405,
      "reused": true,
      "local_path": "papers/repair/4b8cf107df7118e0ffe8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2212.08073",
      "bytes": 2088111,
      "reused": true,
      "local_path": "papers/repair/b3e647be3bc180f47791.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2212.10560",
      "bytes": 4335548,
      "reused": false,
      "local_path": "papers/repair/10d63a1e2d2196f929fc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2211.05100",
      "bytes": 3029243,
      "reused": false,
      "local_path": "papers/repair/101352537283e6ebb422.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2205.01068",
      "bytes": 1765159,
      "reused": false,
      "local_path": "papers/repair/ca8091fa24e3679f3dfb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2205.05131",
      "bytes": 909320,
      "reused": false,
      "local_path": "papers/repair/ce0ad46cab1c8ea63c82.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2201.08239",
      "bytes": 1509689,
      "reused": false,
      "local_path": "papers/repair/2989b90c4b5d7c167587.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2211.01786",
      "bytes": 2577199,
      "reused": false,
      "local_path": "papers/repair/9d979e8024f5772e250d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2209.14375",
      "bytes": 3948958,
      "reused": false,
      "local_path": "papers/repair/7d01b6a4f262c89d30f5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2208.03188",
      "bytes": 13486314,
      "reused": false,
      "local_path": "papers/repair/e509e239a540b3e77736.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2007.06225",
      "bytes": 12922025,
      "reused": false,
      "local_path": "papers/repair/ae6f39fbbc7835156a81.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 25622,
  "expected_body_images": 21,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/d2de6749892a/pages/source.html",
  "local_html": "attachments/d2de6749892a/pages/article.html",
  "body_base_path": "attachments/d2de6749892a/pages/",
  "body_markdown_path": "attachments/d2de6749892a/pages/article.md"
}