{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Sebastian Raschka / Ahead of AI",
  "title_listed": "LLM Training: RLHF and Its Alternatives",
  "keywords": "RLHF,对齐,强化学习",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://magazine.sebastianraschka.com/p/llm-training-rlhf-and-its-alternatives",
  "publish_date": "2023-09",
  "corpus_stage": "对齐/RLHF",
  "notes": "RLHF及其替代方案",
  "id": "540a87128627",
  "collected_at": "2026-10-01T20:32:54",
  "title": "LLM Training: RLHF and Its Alternatives",
  "body_markdown": "I frequently reference a process called Reinforcement Learning with Human Feedback (RLHF) when discussing LLMs, whether in the research news or tutorials. RLHF is an integral part of the modern LLM training pipeline due to its ability to incorporate human preferences into the optimization landscape, which can improve the model's helpfulness and safety.\nIn this article, I will break down RLHF in a step-by-step manner to provide a reference for understanding its central idea and importance. Following up on the previous Ahead of AI article that featured Llama 2, this article will also include a comparison between ChatGPT's and Llama 2's way of doing RLHF.\nFinally, for those wondering about the relevance or necessity of RLHF, I also added a section highlighting the most recent alternatives — I intend to update this section regularly.\nIn short, the table of contents of this article is as follows:\nThe Canonical LLM Training Pipeline\nReinforcement Learning with Human Feedback (RLHF)\nRLHF in Llama 2\nRLHF Alternatives\nThe Canonical LLM Training Pipeline\nModern transformer-based LLMs, such as ChatGPT or Llama 2, undergo a 3-step training procedure:\nPretraining\nSupervised finetuning\nAlignment\nInitially, in the pretraining phase, the models absorb knowledge from enormous unlabeled text datasets. The subsequent supervised finetuning refines these models to better adhere to specific instructions. Lastly, the alignment stage hones the LLMs to respond more helpfully and safely to user prompts.\nNote that this training pipeline is based on\nOpenAI's InstructGPT paper\n, which details the process for GPT-3. This process is widely considered to be the approach behind ChatGPT. Later, we will also compare this method with\nMeta AI's latest Llama 2\nmodel.\nLet's begin with the initial step,\npretraining\n, as depicted below.\nIllustration of the LLM pretraining step\nPretraining typically occurs on a vast text corpus comprising billions to trillions of tokens. In this context, we employ a straightforward next-word prediction task in which the model predicts the subsequent word (or token) from a provided text.\nIf you're unfamiliar with this next-word prediction task, you might like my earlier article\nUnderstanding Encoder And Decoder LLMs\n:\nUnderstanding Encoder And Decoder LLMs\nSebastian Raschka, PhD\n·\nJune 17, 2023\nRead full story\nOne point worth emphasizing is that this type of pretraining allows us to leverage large, unlabeled datasets. As long as we can use the data without infringing on copyrights or disregarding creator preferences, we can access large datasets without the need for manual labeling. In fact, in this pretraining step, the \"label\" is the subsequent word in the text, which is already part of the dataset itself (hence, this pretraining approach is often called self-supervised learning).\nThe next step is s\nupervised finetuning\n, which is summarized in the figure below.\nFinetuning the pretrained model on instruction data.\nThe supervised finetuning stage involves another round of next-token prediction. However, unlike the preceding pretraining stage, we now work with instruction-output pairs, as depicted in the figure above. In this context, the instruction is the input given to the model (it is sometimes accompanied by an optional input text, depending on the task). The output represents a desired response similar to what we expect the model to produce.\nTo provide a concrete example, let's consider the following instruction-output pair:\nInstruction: \"Write a limerick about a pelican.\"\nOutput: \"There once was a pelican so fine...\"\nThe model takes the instruction text (\"Write a limerick about a pelican\") as input and carries out next-token prediction for the output text (\"There once was a pelican so fine...\").\nWhile both employ a similar next-token training objective, supervised finetuning typically uses much smaller datasets than pretraining. This is because it requires instruction-output pairs rather than just raw text. To compile such a dataset, a human (or another high-quality LLM) must write the desired output given a specific instruction — creating such a dataset is a lot of work.\nFollowing this supervised finetuning stage, there's another finetuning phase commonly regarded as the \"alignment\" step, as its primary objective is to align the LLM with human preferences. This is where RLHF comes into play.\nAnnotated graph from InstructGPT paper,\nhttps://arxiv.org/abs/2203.02155\nIn the upcoming section, we will dive into the RLHF-based alignment step. However, for those curious about how it compares to the pretrained base model and the model from supervised finetuning in step 2, I included a chart (above) from the InstructGPT paper, which is the paper that popularized this procedure.\nThe chart above compares a 175B GPT-3 model after supervised finetuning (faint dotted line) with the other method. We can see the base GPT-3 model at the bottom of this chart.\nIf we consider a prompting approach where we query it multiple times and select the best response each time (\"GPT-3 + prompting\"), we can see a somewhat improved performance over the base model (\"GPT-3\"), which is to be expected.\nAdding supervised finetuning to the GPT-3 base model makes the performance (\"GPT-3 + supervised finetuning\") even better than \"GPT-3 + prompting\".\nThe best performance, however, can be obtained from the GPT-3 model after supervised finetuning and RLHF (\"GPT-3 + supervised finetuning + RLHF\") — the two graphs at the top of the chart. (Note that there are two lines at the top of this chart since the researchers experimented with two different sampling procedures.)\nThe next section will describe this RLHF step in more detail.\nReinforcement Learning with Human Feedback (RLHF)\nThe previous section discussed the 3-step training procedure behind modern LLMs like ChatGPT and Llama-2-chat. In this section, we will look at the finetuning stages in more detail, focusing on the RLHF part.\nThe RLHF pipeline takes a pretrained model and finetunes it in a supervised fashion (step 2 in the previous section) and further aligns it with proximal policy optimization (step 3 in the previous section).\nFor simplicity, we will look at the RLHF pipeline in three separate steps:\nRLHF Step 1: Supervised finetuning of the pretrained model\nRLHF Step 2: Creating a reward model\nRLHF Step 3: Finetuning via proximal policy optimization\nRLHF Step 1, shown below, is a supervised finetuning step to create the base model for further RLHF finetuning.\nAnnotated figure from InstructGPT paper,\nhttps://arxiv.org/abs/2203.02155\nIn RLHF step 1, we create or sample prompts (from a database, for example) and ask humans to write good-quality responses. We then use this dataset to finetune the pretrained base model in a supervised fashion.\nNote that this RLHF Step 1 is similar to step 2 in the previous section, \"The Canonical LLM Training Pipeline\". I am listing it here again because it's integral to RLHF.\nIn RLHF Step 2, we then use this model from supervised finetuning to create a reward model, as shown below.\nAnnotated figure from InstructGPT paper,\nhttps://arxiv.org/abs/2203.02155\nAs depicted in the figure above, for each prompt, we generate four to nine responses from the finetuned LLM created in the prior step. An individual then ranks these responses based on their preference. Although this ranking process is time-consuming, it might be somewhat less labor-intensive than creating the dataset for supervised finetuning. This is because ranking responses is likely simpler than writing them.\nUpon compiling a dataset with these rankings, we can design a reward model that outputs a reward score for the optimization subsequent stage in RLHF Step 3. This reward model generally originates from the LLM created in the prior supervised finetuning step. Let's refer to te reward model as RM and the LLM from the supervised finetuning step as SFT. To turn the model from RLHF Step 1 into a reward model, its output layer (the next-token classification layer) is substituted with a regression layer, which features a single output node.\nIf you are interested,\nhere\nis a short video lecture on changing a classification model to a regression model in Unit 4.5 of my Deep Learning Fundamentals course.\nThe third step in the RLHF pipeline is to use the reward (RM) model to finetune the previous model from supervised finetuning (SFT), which is illustrated in the figure below.\nAnnotated figure from InstructGPT paper,\nhttps://arxiv.org/abs/2203.02155\nIn RLHF Step 3, the final stage, we are now updating the SFT model using proximal policy optimization (PPO) based on the reward scores from the reward model we created in RLHF Step 2.\nMore details about PPO are out of the scope of this article, but interested readers can find the mathematical details in these four papers that predate the InstructGPT paper:\n(1)\nAsynchronous Methods for Deep Reinforcement Learning\n(2016) by Mnih, Badia, Mirza, Graves, Lillicrap, Harley, Silver, and Kavukcuoglu introduces policy gradient methods as an alternative to Q-learning in deep learning-based RL.\n(2)\nProximal Policy Optimization Algorithms\n(2017) by Schulman, Wolski, Dhariwal, Radford, and Klimov presents a modified proximal policy-based reinforcement learning procedure that is more data-efficient and scalable than the vanilla policy optimization algorithm above.\n(3)\nFine-Tuning Language Models from Human Preferences\n(2020) by Ziegler, Stiennon, Wu, Brown, Radford, Amodei, Christiano, Irving illustrates the concept of PPO and reward learning to pretrained language models including KL regularization to prevent the policy from diverging too far from natural language.\n(4)\nLearning to Summarize from Human Feedback\n(2020) by Stiennon, Ouyang, Wu, Ziegler, Lowe, Voss, Radford, Amodei, Christiano introduces the popular RLHF three-step procedure that was later also used in the\nInstructGPT paper\n.\nRLHF in Llama 2\nIn the previous section, we looked at the RLHF procedure described in OpenAI's InstructGPT paper. This method is commonly referenced as the one employed to develop ChatGPT. But how does it stack up against Meta AI's recent Llama 2 model?\nMeta AI utilized RLHF in creating the Llama-2-chat models as well. Nevertheless, there are several distinctions between the two approaches, which I've highlighted in the annotated figure below.\nAnnotated figure from Llama 2: Open Foundation and Fine-Tuned Chat Models,\nhttps://arxiv.org/abs/2307.09288\nIn summary, Llama-2-chat follows the same supervised finetuning step on instruction data as InstructGPT in\nRLHF Step 1\n. Yet, during\nRLHF Step 2\n, two reward models are created instead of just one. Furthermore, the Llama-2-chat model evolves through multiple stages, with reward models being updated based on the emerging errors from the Llama-2-chat model. There's also an added rejection sampling step.\nMargin Loss\nAnother distinction not depicted in the above-mentioned annotated figure relates to how model responses are ranked to generate the reward model. In the standard InstructGPT approach for RLHF PPO discussed previously, the researchers collect responses that rank 4-9 outputs from which the researchers create “k choose 2” comparisons.\nFor example, if a human labeler ranks four responses (A-D), such as A < C < D < B, this yields “4 choose 2” = 6 comparisons:\nA < C\nA < D\nA < B\nC < D\nC < B\nD < B\nSimilarly, Llama 2’s dataset is based on binary comparisons of responses, like A < B. However, it appears that each human labeler was only presented 2 responses (as opposed to 4-9 responses) per labeling round.\nMoreover, what’s new is that alongside each binary rank, a \"margin\" label (ranging from \"significantly better\" to \"negligibly better\") is gathered, which can optionally be used in the binary ranking loss via an additional margin parameter to calculate the gap between the two responses.\nWhile InstructGPT used the following cross entropy-based ranking loss to train the reward model:\n\\(\\mathcal{L}_{\\text {ranking }}=-\\log \\left(\\sigma\\left(r_\\theta\\left(x, y_c\\right)-r_\\theta\\left(x, y_r\\right)\\right)\\right)\\)\nLlama 2 added the the margin “m(r)” as a discrete function of the preference rating as follows:\n\\(\\mathcal{L}_{\\text {ranking }}=-\\log \\left(\\sigma\\left(r_\\theta\\left(x, y_c\\right)-r_\\theta\\left(x, y_r\\right)-m(r)\\right)\\right)\\)\nwhere\nr_θ(x,y)\nis the scalar score output for prompt\nx\nand the generated response\ny;\nθ\nare the model weights;\nσ is the logistic sigmoid function that converts the layer outputs to scores ranging from 0 to 1;\ny_c\nis the preferred response chosen by the human annotators;\ny_r\nis the rejected response chosen by the human annotators.\nFor instance, returning a higher margin via “m(r)” will make the difference between the reward of the preferred and rejected responses smaller, resulting in a larger loss, which in turn results in larger gradients, and consequently model changes, during the policy gradient update.\nTwo reward models\nAs mentioned earlier, there are two reward models in Llama 2 instead of one. One reward model is based on\nhelpfulness\n, and the other is based on\nsafety\n. The final reward function that is then used for the model optimization is a linear combination of the two scores.\nExplanation of the Llama 2 ranking method and reward model creation based on an annotated figure from the InstructGPT paper,\nhttps://arxiv.org/abs/2203.02155\nRejection sampling\nMoreover, the authors of Llama 2 employ a training pipeline that iteratively produces multiple RLHF models (from RLHF-V1 to RLHF-V5). Instead of solely relying on the RLHF with PPO method we discussed earlier, they employ two algorithms for RLHF finetuning: PPO and\nrejection sampling\n.\nIn rejection sampling, K outputs are drawn, and the one with the highest reward is chosen for the gradient update during the optimization step, as illustrated below.\nExplanation of the Llama 2 rejection sampling step for creating multiple responses and selecting those with the highest reward based on a figure from the InstructGPT paper\nhttps://arxiv.org/abs/2203.02155\nRejection sampling serves to select samples that has a high reward score in each iteration. As a result, the model undergoes finetuning with samples of a higher reward compared to PPO, which updates based on a single sample each time.\nAfter the initial phases of supervised finetuning, models are exclusively trained using rejection sampling, before later combining both rejection sampling and PPO.\nThe researchers plotted the model performance over the RLHF stages, which shows that the RLHF-finetuned models improve on both the harmlessness and helpfulness axes.\nAnnotated figure from Llama 2: Open Foundation and Fine-Tuned Chat Models,\nhttps://arxiv.org/abs/2307.09288\nNote that the researchers used PPO in the final step, following up on the previous models updated via rejection sampling. As the \"RLHF-v5 (with PPO)\" and \"RLHF-v5 (no PPO)\" comparison in the chart above shows, a model trained with PPO in the final stage is better than a model trained with just rejection sampling. (\nPersonally, I'd be curious to see how it compares to a model finetuned with just PPO and no rejection sampling.\n)\nRLHF Alternatives\nNow that we have discussed and defined the RLHF process, a pretty elaborate procedure, one might wonder whether it's even worth the trouble. The previous graphs from the InstructGPT and Llama 2 papers (shown again below) provide evidence that RLHF is worth the trouble.\nHowever, a lot of ongoing research focuses on developing more efficient alternatives. The most interesting approaches are summarized below.\n(1) Constitutional AI: Harmlessness from AI Feedback\n(Dec 2022,\nhttps://arxiv.org/abs/2212.08073\n)\nIn this Constitutional AI paper, researchers propose a self-training mechanism based on a list of rules humans provide. Similar to the InstructGPT paper mentioned earlier, the proposed method uses a reinforcement learning approach\nFigure from Constitutional AI: Harmlessness from AI Feedback,\nhttps://arxiv.org/abs/2212.08073\nThe term \"\nred teaming\n\" that the researchers use in the figure above refers to a testing approach that has its roots in Cold War military exercises, where\nthe term originally referred to\na group playing the role of the Soviet Union to test US strategies and defenses.\nIn the context of cybersecurity of AI research, the term \"red teaming\" is now used to describe a process where external or internal experts emulate potential adversaries to challenge, test, and ultimately improve the given systems of interest by mimicking the tactics, techniques, and procedures of real-world attackers.\n(2) The Wisdom of Hindsight Makes Language Models Better Instruction Followers\n(Feb 2023,\nhttps://arxiv.org/abs/2302.05206\n)\nThe Wisdom of Hindsight makes Language Models Better Instruction Followers shows that supervised approaches to LLM finetuning can indeed work well. Here, researchers propose a relabeling-based supervised approach for finetuning that outperforms RLHF on 12 BigBench tasks.\nHow does the proposed HIR (Hindsight Instruction Labeling) work? In a nutshell, the method HIR consists of two steps, sampling and training. In the sampling step, prompts and instructions are fed to the LLM to collect the responses. Based on an alignment score, the instruction is relabeled where appropriate in the training phase. Then, the relabeled instructions and the original prompts are used to finetune the LLM. Using this relabeling approach, the researchers effectively turn failure cases (cases where the LLM creates outputs that don't match the original instructions) into useful training data for supervised learning.\nAnnotated figure from\nThe Wisdom of Hindsight Makes Language Models Better Instruction Followers\n,\nhttps://arxiv.org/abs/2302.05206\nNote that this study is not directly comparable to the RLHF work in InstructGPT, for example, since it seems to be using heuristics (\"However, as most human-feedback data is hard to collect, we adopt a scripted feedback function ...\"). The results of the HIR hindsight approach are still very compelling, though.\n(3) Direct Preference Optimization: Your Language Model is Secretly a Reward Model\n(\nhttps://arxiv.org/abs/2305.18290\n, May 2023)\nDirect Preference Optimization (DPO) is an alternative to RLHF with PPO where the researchers show that the cross entropy loss for fitting the reward model in RLHF can be used directly to finetune the LLM. According to their benchmarks, it's more efficient to use DPO and often also preferred over RLHF/PPO in terms of response quality.\nAnnotated figure from\nhttps://arxiv.org/abs/2305.18290\n(4) Contrastive Preference Learning: Learning from Human Feedback without RL\n(Oct 2023,\nhttps://arxiv.org/abs/2310.13639\n)\nLike Direct Preference Optimization (DPO), Contrastive Preference Learning (CPL) is an approach to simplify RLHF by eliminating the reward model learning. Like DPO, CPL uses a supervised learning objective, specifically, a contrastive loss. (In the paper's appendix, the authors show that DPO is a special case of CPL.) While the experiments were based on a robotics environment, CPL could also be applied to LLM finetuning.\n(5) Reinforced Self-Training (ReST) for Language Modeling\n(Aug 2023,\nhttps://arxiv.org/abs/2308.08998\n)\nReST is an alternative to reinforcement learning with human feedback (RLHF) that aligns LLMs with human preferences. ReST uses a sampling approach to create an improved dataset, iteratively training on increasingly higher-quality subsets to refine its reward function. According to the authors, ReST achieves greater efficiency compared to standard online RLHF methods (like RLHF with proximal policy optimization, PPO) by generating its training dataset offline, but a comprehensive comparison to standard RLHF PPO methods as used in InstructGPT or Llama 2 is missing\nAnnotated figure from\nhttps://arxiv.org/abs/2308.08998\n(6) RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback\n(Sep 2023,\nhttps://arxiv.org/abs/2309.00267\n)\nThe recent reinforcement learning with AI feedback (RLAIF) study shows that the ratings for the reward model training in RLHF don't necessarily have to be provided by humans but can be generated by an LLM (here: PaLM 2). Human evaluators prefer the RLAIF model half of the time over traditional RLHF models, which means that they actually don't prefer one model over the other.\nAn additional interesting side note is that both RLHF and RLAIF strongly outperformed models that are purely trained via supervised instruction finetuning.\nAnnotated figure from the RLAIF paper,\nhttps://arxiv.org/abs/2309.00267\nThe outcome of this study is very useful and interesting since it basically means that we might be able to make RLHF-based training more efficient and accessible. However, it remains to be seen how these RLAIF models perform in qualitative studies that focus on safety and truthfulness of the information content, which is only partially captured by human preferences studies.\nFinal Remarks\nWhether these alternatives will be worthwhile in practice remains to be seen as there is currently no true competitor to Llama 2 and Code Llama-scale models that have been trained without RLHF.\nAlso, if you have tried any of these methods or stumbled upon additional promising approaches, I'd love to hear!\nThis magazine is a personal passion project that does not offer direct compensation. However, for those who wish to support me, please consider purchasing a copy of\none of my books\n. If you find them insightful and beneficial, please feel free to recommend them to your friends and colleagues.\nMachine Learning with PyTorch and Scikit-Learn\n,\nMachine Learning Q and AI\n, and\nBuild a Large Language Model (from Scratch)\nYour support means a great deal! Thank you!",
  "metadata": {
    "title": "LLM Training: RLHF and Its Alternatives",
    "description": "I frequently reference a process called Reinforcement Learning with Human Feedback (RLHF) when discussing LLMs, whether in the research news or tutorials.",
    "keywords": null,
    "author": "Sebastian Raschka, PhD",
    "og:type": "article",
    "og:title": "LLM Training: RLHF and Its Alternatives",
    "og:description": "I frequently reference a process called Reinforcement Learning with Human Feedback (RLHF) when discussing LLMs, whether in the research news or tutorials.",
    "og:image": "https://substackcdn.com/image/fetch/$s_!kAkh!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe06a33c8-cdbd-4f5e-8380-86fb71a075c8_2216x1232.png",
    "og:url": "https://magazine.sebastianraschka.com/p/llm-training-rlhf-and-its-alternatives",
    "twitter:title": "LLM Training: RLHF and Its Alternatives",
    "twitter:description": "I frequently reference a process called Reinforcement Learning with Human Feedback (RLHF) when discussing LLMs, whether in the research news or tutorials.",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!Cuse!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fsebastianraschka.substack.com%2Fapi%2Fv1%2Fpost_preview%2F136862014%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2024-08-12T13:17:46.203Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!732u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52beec42-4ba4-4477-80d3-4dbb8f3162d3_1600x950.png",
      "bytes": 123389,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-152910f7800615ac0a99.png",
      "src": "https://substackcdn.com/image/fetch/$s_!732u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52beec42-4ba4-4477-80d3-4dbb8f3162d3_1600x950.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!YVUm!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1655aff8-c9bd-4a93-9e72-f1911359a667_1646x1090.jpeg",
      "bytes": 3498,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-afa47476bee5dc09ce97.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!YVUm!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1655aff8-c9bd-4a93-9e72-f1911359a667_1646x1090.jpeg",
      "alt": "Understanding Encoder And Decoder LLMs",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!2-bw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97601e27-cee1-443d-b9ac-a5ea6f802fc9_2156x1164.png",
      "bytes": 84743,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-6b443281246093c2c10d.png",
      "src": "https://substackcdn.com/image/fetch/$s_!2-bw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97601e27-cee1-443d-b9ac-a5ea6f802fc9_2156x1164.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!7eSR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c3c20d4-8c22-4203-9ea2-0da80e7eabb5_2088x1178.png",
      "bytes": 74733,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-37a6ee97ac0564972d10.png",
      "src": "https://substackcdn.com/image/fetch/$s_!7eSR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c3c20d4-8c22-4203-9ea2-0da80e7eabb5_2088x1178.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!6nM7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce4819d8-db35-4218-8570-8b2412b59934_1600x681.png",
      "bytes": 84497,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-075edd6c86eb1325c421.png",
      "src": "https://substackcdn.com/image/fetch/$s_!6nM7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce4819d8-db35-4218-8570-8b2412b59934_1600x681.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!rEYg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41432907-13e3-48d3-ad8f-9db5baa76dfd_1600x778.png",
      "bytes": 125590,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-25e9436e276720426b85.png",
      "src": "https://substackcdn.com/image/fetch/$s_!rEYg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41432907-13e3-48d3-ad8f-9db5baa76dfd_1600x778.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!nC2U!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89309fc6-320e-431c-ab2d-110cd1461f0e_1600x857.png",
      "bytes": 130325,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-2abc02172690e58c6210.png",
      "src": "https://substackcdn.com/image/fetch/$s_!nC2U!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89309fc6-320e-431c-ab2d-110cd1461f0e_1600x857.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!MVjs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feef9ddfa-657d-4784-9e87-371e7617143d_1600x768.png",
      "bytes": 70771,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-3597dffd0b92a3cbab03.png",
      "src": "https://substackcdn.com/image/fetch/$s_!MVjs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feef9ddfa-657d-4784-9e87-371e7617143d_1600x768.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!aso9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17d06b15-e1de-49df-907c-65dc26d1fc21_1444x986.png",
      "bytes": 240520,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-fe7c9e7890c17ff46eab.png",
      "src": "https://substackcdn.com/image/fetch/$s_!aso9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17d06b15-e1de-49df-907c-65dc26d1fc21_1444x986.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!KJK0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd187fa03-982c-4818-bbd7-833d2ef10386_1298x864.png",
      "bytes": 130463,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-522a43ee6c29d364ca41.png",
      "src": "https://substackcdn.com/image/fetch/$s_!KJK0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd187fa03-982c-4818-bbd7-833d2ef10386_1298x864.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!c2YN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F334cb151-1c0d-45e0-ad66-ef4509d022c7_804x748.png",
      "bytes": 29783,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-6b44cc04ea59c62f1642.png",
      "src": "https://substackcdn.com/image/fetch/$s_!c2YN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F334cb151-1c0d-45e0-ad66-ef4509d022c7_804x748.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Mh7P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6727c2d5-e824-4361-a938-b05f494a97f4_1600x761.png",
      "bytes": 167751,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-661e9b05419fc1983db9.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Mh7P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6727c2d5-e824-4361-a938-b05f494a97f4_1600x761.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!rnnt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6daf523-60c3-4c86-abe6-ca71e83d838f_1600x700.png",
      "bytes": 73181,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-6220119d5323aeacf681.png",
      "src": "https://substackcdn.com/image/fetch/$s_!rnnt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6daf523-60c3-4c86-abe6-ca71e83d838f_1600x700.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!cy8S!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46df147c-6afe-4c32-86d9-1ad0fd6898c4_1544x1432.png",
      "bytes": 156109,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-0e53304ff330be33b1d5.png",
      "src": "https://substackcdn.com/image/fetch/$s_!cy8S!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46df147c-6afe-4c32-86d9-1ad0fd6898c4_1544x1432.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!3Zrz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F071f6b8c-89b4-4920-ad89-1f5b0dfba385_1358x1084.png",
      "bytes": 111852,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-946514fae1a9ab4ecd7b.png",
      "src": "https://substackcdn.com/image/fetch/$s_!3Zrz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F071f6b8c-89b4-4920-ad89-1f5b0dfba385_1358x1084.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!zqLh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F481cac18-c77e-4dc4-898f-355492c5eae6_1322x736.png",
      "bytes": 122693,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-ab61b610f40e06ce795d.png",
      "src": "https://substackcdn.com/image/fetch/$s_!zqLh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F481cac18-c77e-4dc4-898f-355492c5eae6_1322x736.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!-s2u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49781970-bcb7-43a6-94ee-d9b3b1e155d4_1358x1194.png",
      "bytes": 115148,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-297171c4f52337b17d64.png",
      "src": "https://substackcdn.com/image/fetch/$s_!-s2u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49781970-bcb7-43a6-94ee-d9b3b1e155d4_1358x1194.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!CxIZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27b6d27d-1052-4a4f-a419-a69cdd36704a_1118x454.png",
      "bytes": 77779,
      "reused": false,
      "local_path": "attachments/540a87128627/images/body-4095da258ac44040a4a6.png",
      "src": "https://substackcdn.com/image/fetch/$s_!CxIZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27b6d27d-1052-4a4f-a419-a69cdd36704a_1118x454.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://arxiv.org/abs/2203.02155",
      "label": "OpenAI's InstructGPT paper",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2307.09288",
      "label": "Meta AI's latest Llama 2",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!732u!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52beec42-4ba4-4477-80d3-4dbb8f3162d3_1600x950.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://magazine.sebastianraschka.com/p/understanding-encoder-and-decoder",
      "label": "Understanding Encoder And Decoder LLMs Sebastian Raschka, PhD · June 17, 2023 Read full story",
      "type": "a"
    },
    {
      "href": "https://substack.com/profile/27393275-sebastian-raschka-phd",
      "label": "Sebastian Raschka, PhD",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!7eSR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4c3c20d4-8c22-4203-9ea2-0da80e7eabb5_2088x1178.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!rEYg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41432907-13e3-48d3-ad8f-9db5baa76dfd_1600x778.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://lightning.ai/courses/deep-learning-fundamentals/training-multilayer-neural-networks-overview/4-5-multilayer-neural-networks-for-regression-parts-1-2/",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!nC2U!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89309fc6-320e-431c-ab2d-110cd1461f0e_1600x857.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1602.01783",
      "label": "Asynchronous Methods for Deep Reinforcement Learning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1707.06347",
      "label": "Proximal Policy Optimization Algorithms",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1909.08593",
      "label": "Fine-Tuning Language Models from Human Preferences",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.01325",
      "label": "Learning to Summarize from Human Feedback",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!MVjs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feef9ddfa-657d-4784-9e87-371e7617143d_1600x768.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!aso9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17d06b15-e1de-49df-907c-65dc26d1fc21_1444x986.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2204.05862",
      "label": "rejection sampling",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!KJK0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd187fa03-982c-4818-bbd7-833d2ef10386_1298x864.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!c2YN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F334cb151-1c0d-45e0-ad66-ef4509d022c7_804x748.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!Mh7P!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6727c2d5-e824-4361-a938-b05f494a97f4_1600x761.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2212.08073",
      "label": "https://arxiv.org/abs/2212.08073",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!rnnt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6daf523-60c3-4c86-abe6-ca71e83d838f_1600x700.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Red_team",
      "label": "red teaming",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2302.05206",
      "label": "https://arxiv.org/abs/2302.05206",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!cy8S!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46df147c-6afe-4c32-86d9-1ad0fd6898c4_1544x1432.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.18290",
      "label": "https://arxiv.org/abs/2305.18290",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!3Zrz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F071f6b8c-89b4-4920-ad89-1f5b0dfba385_1358x1084.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2310.13639",
      "label": "https://arxiv.org/abs/2310.13639",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2308.08998",
      "label": "https://arxiv.org/abs/2308.08998",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!zqLh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F481cac18-c77e-4dc4-898f-355492c5eae6_1322x736.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2309.00267",
      "label": "https://arxiv.org/abs/2309.00267",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!-s2u!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49781970-bcb7-43a6-94ee-d9b3b1e155d4_1358x1194.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://sebastianraschka.com/books",
      "label": "one of my books",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!CxIZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27b6d27d-1052-4a4f-a419-a69cdd36704a_1118x454.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Machine-Learning-PyTorch-Scikit-Learn-scikit-learn-ebook-dp-B09NW48MR1/dp/B09NW48MR1/",
      "label": "Machine Learning with PyTorch and Scikit-Learn",
      "type": "a"
    },
    {
      "href": "https://nostarch.com/machine-learning-and-ai-beyond-basics",
      "label": "Machine Learning Q and AI",
      "type": "a"
    },
    {
      "href": "http://mng.bz/M96o",
      "label": "Build a Large Language Model (from Scratch)",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:26Z",
  "source_url": "https://magazine.sebastianraschka.com/p/llm-training-rlhf-and-its-alternatives",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2203.02155",
      "bytes": 1797405,
      "reused": true,
      "local_path": "papers/repair/4b8cf107df7118e0ffe8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2307.09288",
      "bytes": 13661300,
      "reused": true,
      "local_path": "papers/repair/01c99984e3c41b2873f5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1602.01783",
      "bytes": 2302720,
      "reused": true,
      "local_path": "papers/repair/4ebd69e62e1319a151ba.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1707.06347",
      "bytes": 2923532,
      "reused": true,
      "local_path": "papers/repair/5c73fa759d96331836bf.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1909.08593",
      "bytes": 967470,
      "reused": true,
      "local_path": "papers/repair/6e863e9d0bbb0e25e44f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.01325",
      "bytes": 1670544,
      "reused": true,
      "local_path": "papers/repair/2b4f670aa5138d617d3b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2204.05862",
      "bytes": 9503688,
      "reused": true,
      "local_path": "papers/repair/9b44d30c07a465bc130a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2212.08073",
      "bytes": 2088111,
      "reused": true,
      "local_path": "papers/repair/b3e647be3bc180f47791.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2302.05206",
      "bytes": 1287506,
      "reused": false,
      "local_path": "papers/repair/5a56111011d6bdf875f7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.18290",
      "bytes": 1299212,
      "reused": false,
      "local_path": "papers/repair/ba63f83d413059cb8722.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2310.13639",
      "bytes": 1398683,
      "reused": false,
      "local_path": "papers/repair/e7f44b5ee9d978a2a0e9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2308.08998",
      "bytes": 1404121,
      "reused": false,
      "local_path": "papers/repair/41a987b8a81e377765dc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2309.00267",
      "bytes": 2531894,
      "reused": false,
      "local_path": "papers/repair/34c41ca57840163efcb7.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 21767,
  "expected_body_images": 18,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/540a87128627/pages/source.html",
  "local_html": "attachments/540a87128627/pages/article.html",
  "body_base_path": "attachments/540a87128627/pages/",
  "body_markdown_path": "attachments/540a87128627/pages/article.md"
}