{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Sebastian Raschka / Ahead of AI",
  "title_listed": "Instruction Pretraining LLMs",
  "keywords": "指令预训练,SFT,数据合成",
  "domain": "机器学习",
  "content_type": "研究综述",
  "is_paid_flag": "否",
  "url": "https://magazine.sebastianraschka.com/p/instruction-pretraining-llms",
  "publish_date": "2024-09",
  "corpus_stage": "SFT",
  "notes": "指令预训练方法对比，含数据质量与配比分析",
  "id": "ef29388bfc9d",
  "collected_at": "2026-10-01T19:49:21",
  "title": "Instruction Pretraining LLMs - by Sebastian Raschka, PhD",
  "body_markdown": "A lot has happened last month: Apple announced the integration of on-device LLMs, Nvidia shared their large Nemotron model, FlashAttention-3 was announced, Google's Gemma 2 came out, and much more.\nYou've probably already read about it all in various news outlets. So, in this article, I want to focus on recent research centered on instruction finetuning, a fundamental technique for training LLMs.\nWhat I am going to cover in this article:\nA new, cost-effective method for generating data for instruction finetuning\nInstruction finetuning from scratch\nPretraining LLMs with instruction data\nAn overview of what's new in Gemma 2\nAn overview of all the other interesting research papers that came out in June\nHappy reading!\n1. Creating Alignment Data from Scratch\nThe\nMagpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing\npaper shares a fascinating hack to generate a high-quality dataset for LLM instruction finetuning. While this doesn't offer any particularly recent research insights, it's one of those interesting, practical exploits that seems super useful.\n1.1 Generating An Instruction Dataset From Nothing\nWhat distinguishes this instruction-data-generating method from others is that it can be fully automated and doesn't require any initial questions or instructions. As the paper title suggests, it enables the creation of an instruction dataset from \"Nothing\" – the only thing we need is a locally running Llama 3 8B model. The figure below summarizes how this method works.\nAnnotated illustration of the Magpie method for generating a synthetic dataset for instruction finetuning. The figure is based on illustrations from the Magpie paper: https://arxiv.org/abs/2406.08464\nEssentially, as shown in the figure above, we just have to prompt the\nLlama 3 8B Instruct\nmodel with a pre-query template, and it will generate an instruction for us. Then, we feed that instruction back to the LLM, and it will generate a response. If we repeat this procedure a couple of thousand times, we obtain a dataset for instruction finetuning. (Optionally, we can apply an LLM to filter the instruction-response pairs by quality.)\n1.2 Dataset quality\nWhat's fascinating is that with the resulting instruction dataset, the authors found that finetuning a Llama 3 8B base model with just instruction finetuning (no preference finetuning via RLHF and DPO) beats the original Llama 2 8B Instruct model by Meta AI, as shown in the figure below.\nA Llama 3 8B base model finetuned on the Magpie-generated instruction dataset beats the original Llama 3 8B Instruct model. Based on an annotated illustration from the Magpie paper: https://arxiv.org/abs/2406.08464\nThe Magpie results shown in the figure above wer achieved with 300 thousand samples only. In comparison, The original Llama 3 Instruct model was finetuned and aligned on 100 million samples!\n1.3 Running the Dataset Generation Locally\nI was skeptical at first, so I tried to implement this myself. It really works!\nHere\n, you can find my reimplementation using Ollama, which even runs fine locally on a MacBook Air.\nCode screenshot from a reimplementation of the Magpie method that runs locally. The code is available\nhere\n.\n1.4 Additional Details\nThe authors created two sets of datasets: A \"Pro\" version using the Llama 3 70B Instruct model and an \"Air\" version using the Llama 3 8B Instruct model. As an earlier figure showed, the Magpie-Pro-generated dataset results in slightly stronger models compared to the Magpie-Air dataset when using it to instruction-finetune a Llama 3 8B base model.\nThe figure below shows an additional comparison of the dataset qualities and difficulties as rated via an LLM.\nAnnotated plots from the Magpie paper showing the dataset quality and difficulty of the Air and Pro datasets relative to each other.\nAs the figure above shows, the quality of the Air and Pro datasets is roughly on par. In addition, it would have been interesting to see how the Alpaca dataset compares to these. (The assumption is that the Magpie data is of much higher quality than Alpaca, but a reference point would be interesting.)\nFurthermore, the paper contains an analysis showing that the breadth or diversity in this dataset is much larger than that of other popular datasets for instruction finetuning, such as Alpaca, Evol Instruct, and UltraChat. In addition, when compared to models trained with other instruction finetuning datasets, the Magpie-Pro finetuned model also compares very favorably.\n1.5 Conclusion\nOverall, I think that Magpie is an interesting exploit that is, on the one hand, fascinating in its effectiveness and, on the other hand, has a lot of practical utility. I will certainly consider it as an interesting, simple, and cost-effective candidate for constructing general-purpose instruction datasets in the future.\n2. Instruction Finetuning from Scratch\nIf you are looking for a resource to understand the instruction finetuning process in LLMs, I am happy to share that Chapter 7 on instruction finetuning LLMs is now finally\nlive on the Manning website\n.\nThis is the longest chapter in the book and takes a from-scratch approach to implementing the instruction finetuning pipeline. This includes everything from input formatting to batching with a custom collate function, masking padding tokens, the training loop itself, and scoring the response quality of the finetuned LLM on a custom test set.\n(The exercises include changing prompt styles, instruction masking, and adding LoRA.)\nHappy coding!\nAn overview of chapter 7 in my\n\"Build a Large Language Model From Scratch\"\nbook. Supplementary code materials are available\nhere on GitHub\n.\nPS: it's also the last chapter, and the publisher is currently preparing the layouts for the print version.\n3. Instruction Pretraining LLMs\nIn the paper \"Instruction Pre-Training: Language Models are Supervised Multitask Learners\" (\nhttps://arxiv.org/abs/2406.14491\n), researchers investigate whether LLM pretraining can be made more efficient by including synthetic instruction-response pairs instead of just raw text. (Here, \"raw text\" means text from books, websites, papers, and so forth that has not been reprocessed into a specific format.)\nA comparison between regular pretraining (top) and the proposed instruction pretraining approach (bottom) via an annotated figure from\nhttps://arxiv.org/abs/2406.14491\nSpecifically, the researchers experiment with generating instruction-response data from the raw training corpus itself via an \"instruction synthesizer,\" an LLM specifically finetuned for this task.\n(Note that this is not the first paper proposing the formatting of raw text as instruction data. Another work that comes to mind is \"Genie: Achieving Human Parity in Content-Grounded Datasets Generation\" (\nhttps://arxiv.org/abs/2401.14367\n). I also recall seeing another paper or blog post using instruction data during pretraining a few months ago—I discussed this method with some of my colleagues—but unfortunately, I couldn't find the reference. Nonetheless, the paper discussed here is particularly intriguing since it builds on openly available LLMs that run locally and covers both pretraining and continual pretraining.)\n3.1 Instruction Synthesizer\nBefore we dive into the pretraining and continual pretraining results, let's talk about the core component of this method: the instruction synthesizer. This is an openly available Mistral 7B v0.1 LLM (which I wrote about last year here:\nhttps://magazine.sebastianraschka.com/i/138555764/mistral-b\n) that has been finetuned to generate instruction-response pairs from raw text.\nTo finetune this synthesizer, the researchers use datasets such as HotpotQA (\nhttps://arxiv.org/abs/1809.09600\n), which consists of passages from Wikipedia associated with questions and answers. For this, the authors also ensure that a variety of tasks, like commonsense reasoning, sentiment analysis, math problems, etc., are covered.\nThe input and output data of the instruction synthesizer via an annotated figure from https://arxiv.org/abs/2406.14491\nOnce this instruction synthesizer is developed (i.e., finetuned), it can be used to generate the input data for pretraining the target LLMs.\nOne last noteworthy detail regarding the instruction synthesizer is that multiple raw texts (T\nn\n) and instruction-response pairs (I\nn\n⊕ R\nn\n) are concatenated as few-shot examples, as shown in the figure below.\nThe formatting of the instruction-data for finetuning (and using) the instruction synthesizer via an annotated figure from https://arxiv.org/abs/2406.14491\n3.2 Pretraining with Instruction Data\nNow that we have discussed the method to generate the instruction-response pairs, let's get to the interesting part: how well do models train on this augmented dataset. The first set of results looks at two small models trained from scratch: 500M parameters and 1.3B parameters (both are based on the Mistral architecture).\nA comparison of 3 different pretraining approaches used to train models from scratch (annotated table from https://arxiv.org/abs/2406.14491)\nAs we can see in the table above, the model trained via the proposed instruction pretraining approach (\nInstruct PT\n) performs best on most benchmark tasks (higher values are better).\nNote, though, that it has seen more tokens than the\nVanilla PT\napproach since it included the synthesized instruction-response pairs. Hence, the authors included the\nMix PT\ncomparison, which is a model that has been trained on a data mix containing both the raw text and the instruction data used to train the synthesizer.\nFrom this comparison, we can see that not simply using any instruction data makes the difference. The fact that\nInstruct PT\nperforms better than\nMix PT\non most tasks illustrates that the nature of the instruction-response data (i.e., instruction-response data related to the raw data) makes the difference. (The authors conducted all experiments using the same number of tokens.)\nIn addition, it's worth noting that the Instruct PT pretrained models have another advantage: They improve a more when they are instruction-finetuned afterwards, as the figure below shows.\nFinetuning LLMs that have been pretrained with either the traditional pretraining pardigm (Vanilla PT) or instruction pretraining (annotated figure from https://arxiv.org/abs/2406.14491)\n3.3 Continual Pretraining with Instruction Data\nPretraining from scratch is interesting because that's how LLMs are created in the first place. However, I'd say that practitioners care more about continual pretraining and finetuning.\nContinual pretraining here means that we take an existing pretrained model and pretrain it further on new domain data. For instance, think of a Llama 3 8B base model that has been trained on a general text corpus and that you want to adapt for finance, medical, legal, or other domains.\nThe table below summarizes the results the researchers obtained when applying the instruction pretraining method to a pretrained Llama 3 8B base model. Specifically, they conducted continual pretraining with both biomedical texts and finance texts.\nA comparison of 3 different pretraining approaches used for continual pretraining (annotated table from https://arxiv.org/abs/2406.14491)\nLooking at the table above, we can see that the instruction pretraining approach (\nInstruct PT\n) clearly outperforms the vanilla pretraining (\nVanilla PT\n) approach (here, this means regular continual pretraining of the base model).\nThe Llama 3 70B base model is included as a reference; I suppose to showcase that small specialized models can beat larger general models.\n3.4 Conclusion\nAlmost every time I explain the LLM pretraining pipeline to someone, they are surprised by its simplicity and the fact that this is still what's commonly used to train LLMs today. The instruction pretraining approach is quite refreshing in that sense.\nOne caveat is that for large pretraining corpora, it might still be expensive to create the instruction-augmented corpora. However, the nice thing about generated data is that it can be reused in many different projects once created.\n4. Gemma 2\nI cannot write this article without mentioning Google's new\nGemma 2 models\n, which are arguably the biggest model release last month. However, when it comes to pure size, Nvidia's Nemotron-4 340B takes the crown (https://arxiv.org/abs/2406.11704). The Gemma 2 models come in 2.6B, 9B, and 27B parameter versions.\nSince this article is already quite lengthy, and you're likely familiar with Gemma 2 from other sources, let's cut to the chase. What are the main highlights and noteworthy updates in Google's newly released Gemma 2 LLMs? The main theme is exploring techniques without necessarily increasing the size of training datasets but rather focusing on developing relatively small and efficient LLMs.\nSpecifically, they blend three main architectural and training choices to create the 2.6B and 9B parameter models: sliding window attention, grouped-query attention, and knowledge distillation.\n4.1 Sliding window attention\nSliding window attention\n(e.g., as popularized by Mistral) is a technique using a fixed-sized attention block that allows a current token to attend to only a specific number of previous tokens instead of all previous tokens, as illustrated in the figure below.\nAnnotated figure from\nhttps://arxiv.org/abs/2310.06825\nexplaining sliding window attention.\nIn the case of Gemma 2, the authors alternated between regular attention and sliding window attention layers. The sliding attention block size was 4096 tokens, spanning a total block size of 8192 tokens.\nSliding window attention is mainly used to improve computational performance, and the researchers also included a small ablation study showing that there's a barely noticeable difference in perplexity when shrinking the block size during inference.\nAn ablation study from the\nGemma 2 technical report\nshowing that a decreased block size for the sliding window barely impacts the modeling performance of the 9B parameter model during inference.\n(It would have been interesting to see the GPU memory improvement side-by-side.)\n4.2 Group-query attention\nGroup-query attention\n(like in Llama 2 and 3) can be regarded as a more generalized form of multi-query attention. The motivation behind this is to reduce the number of trainable parameters by sharing the same Keys and Values heads for multiple Query heads, thereby lowering computational requirements.\nAnnotated figure from\nAinslie\net al.\n2023\n4.3 Knowledge distillation\nThe general idea of\nKnowledge distillation\n(as in MiniLLM, https://arxiv.org/abs/2306.08543) is to transfer knowledge from a larger model (the teacher) to a smaller model (the student). Here, they trained a 27B (teacher) model from scratch and then trained the smaller 2B and 9B (student) models on the outputs of the larger teacher model. The 27B model doesn't use knowledge distillation but was trained from scratch to serve as a \"teacher\" for the smaller models.\nAn overview of knowledge distillation from\nmy Machine Learning Q and AI book\nin the context of computer vision. In an LLM-context, think of text instead of images and predicted tokens instead of class labels.\n4.4 Other interesting architecture details\nThe paper contains many other interesting tidbits. For instance, one hallmark of Gemma 2 is its relatively large vocabulary size: 256,000 tokens. This is similar to the first Gemma model, but it's still worth noting since it's twice the size of the Llama 3 vocabulary (128,000) and eight times the size of the Phi-3 vocabulary (32,000).\nThe vocabulary size of an LLM refers to the number of unique tokens (words, subwords, or characters) that the model can recognize and generate.\nA large vocabulary size in LLMs allows for better coverage of words and concepts, improved handling of multilingual content, and reduced tokenization artifacts. However, a large vocabulary size also comes with trade-offs, such as increased model size and potentially slower inference due to the larger embedding and output layers. (That's where the sliding window attention and multi-query attention mechanism are important to offset this.)\nThere's also an interesting section on \"logit capping,\" a technique I haven't seen used before. Essentially, it is a form of min-max normalizing and clipping of the logit values to keep them within a certain range. I presume this is to improve stability and gradient flow during training.\nlogits ← soft_cap ∗ tanh(logits/soft_cap).\nAdditionally, they leverage model merging techniques to combine models from multiple runs with different hyperparameters, although the paper doesn't provide much detail about that. (However, interested readers can read more about this in\nWARP: On the Benefits of Weight Averaged Rewarded Policies\n, which Gemma 2 uses for this.)\nIn terms of modeling performance, Gemma 2 is almost as good as the 3x larger Llama 3 70B, and it beats the old Qwen 1.5 32B model. It would be interesting to see a comparison with the more recent Qwen 2 model.\nA comparison between two other popular models with openly available weights: Llama 3 and Qwen 1.5. (Annotated table from the\nGemma 2 technical report\n).\nPersonally, a highlight is that the Gemma 2 report includes ablation studies for some of its architectural choices. This was once a given in academic research but is increasingly rare for LLM research.\nAn example of one of the ablation studies included in the\nGemma 2 technical report\n. Here \"wide\" refers to a model with 28 layers and an intermediate size of 24,576, and \"deep\" refers to a architecture with 42 layers with an intermediate size of 14,336.\n4.5 Conclusion\nIt's refreshing to see such a relatively detailed technical report from Google. When it comes to the model itself, based on public consensus, Gemma 2 is likely the most capable model for single-GPU use cases today. For larger models, Llama 3 70B and Qwen 2 72B remain strong contenders.\nSupporting Ahead of AI\nAhead of AI is a personal passion project that does not offer direct compensation. However, for those who wish to support me, please consider purchasing a copy of\nmy books\n. If you find them insightful and beneficial, please feel free to recommend them to your friends and colleagues.\nIf you have a few moments, a review of\nMachine Learning Q and AI\nor\nMachine Learning with PyTorch and Scikit-Learn\non Amazon would really help, too!\nYour support means a great deal and is tremendously helpful in continuing this journey. Thank you!\n5. Other Interesting Research Papers In April\nBelow is a selection of other interesting papers I stumbled upon this month. Given the length of this list, I highlighted those 20 I found particularly interesting with an asterisk (*). However, please note that this list and its annotations are purely based on my interests and relevance to my own projects.\nScaling Synthetic Data Creation with 1,000,000,000 Personas\nby Chan, Wang, Yu,\net al.\n(28 June),\nhttps://arxiv.org/abs/2406.20094\nThe research proposes a persona-driven data synthesis methodology that utilizes an LLM to create diverse synthetic data by leveraging a vast collection of automatically curated personas, called Persona Hub, which represents about 13% of the world's population.\nLLM Critics Help Catch LLM Bugs\nby McAleese, Pokorny, Ceron Uribe,\net al.\n(28 June),\nhttps://arxiv.org/abs/2407.00215\nThis study develops \"critic\" models using RLHF to assist humans in evaluating model-generated code, training LLMs to write natural language feedback on code errors, and demonstrating their effectiveness in catching bugs across various tasks.\nDirect Preference Knowledge Distillation for Large Language Models\nby Li, Gu, Dong,\net al.\n(28 June),\nhttps://arxiv.org/abs/2406.19774\nDPKD reformulates Knowledge Distillation for LLMs into a two-stage process: first optimizing an objective combining implicit reward and reverse KL divergence, then improving the preference probability of teacher outputs over student outputs.\nChanging Answer Order Can Decrease MMLU Accuracy\nby Gupta, Pantoja, Ross,\net al.\n(27 June),\nhttps://arxiv.org/abs/2406.19470\nThis study investigates the robustness of accuracy measurements on the MMLU benchmark for LLMs, revealing that shuffling answer label contents leads to decreased accuracy across models, with varying sensitivity.\nFrom Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data\nby Xiong, Papageorgiou, Lee, and Papailiopoulos (27 June),\nhttps://arxiv.org/abs/2406.19292\nThis study proposes a finetuning approach using a synthetic dataset of numerical key-value retrieval tasks to improve LLM's long-context information retrieval and reasoning capabilities.\nDataset Size Recovery from LoRA Weights\nby Salama, Kahana, Horwitz, and Hoshen (27 June),\nhttps://arxiv.org/abs/2406.19395\nThis study introduces a method for recovering the number of images used to finetune a vision model using LoRA, by analyzing the norm and spectrum of LoRA matrices.\nStep-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs\nby Azerbayev, Shao, Lin,\net al.\n(26 June),\nhttps://arxiv.org/abs/2406.18629\nThis paper introduces Step-DPO, a method that optimizes individual reasoning steps in mathematical problem-solving for LLMs using a custom 10K step-wise preference pair dataset.\nRouteLLM: Learning to Route LLMs with Preference Data\nby Ong, Amjad,\net al.\n(26 June),\nhttps://arxiv.org/abs/2406.18665\nThis study proposes efficient router models that dynamically select between stronger and weaker LLMs during inference to optimize cost-performance trade-offs.\n* A Closer Look into Mixture-of-Experts in Large Language Models\nby Zhang, Liu, Patel,\net al.\n(26 June),\nhttps://arxiv.org/abs/2406.18219\nThis study looks at the inner workings of Mixture-of-Experts (MoE) LLMs to share  insights about neuron behavior, expert selection criteria, and expert diversity across layers, while providing practical suggestions for MoE design and implementation based on these observations.\n* Following Length Constraints in Instructions\nby Yuan, Kulikov, Yu,\net al.\n(25 June),\nhttps://arxiv.org/abs/2406.17744\nThis study introduces a method to train LLMs that can follow user-specified length constraints at inference time, addressing the length bias in model evaluation and outperforming standard instruction-following models in length-controlled tasks.\nLongIns: A Challenging Long-context Instruction-based Exam for LLMs\nby Shaham, Bai, An,\net al.\n(25 June),\nhttps://arxiv.org/abs/2406.17588\nLongIns is a new benchmark for evaluating LLM's long-context capabilities, using three settings to assess retrieval and reasoning abilities.\n*\nThe FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale\nby He, Wang, Shen,\net al.\n(25 June),\nhttps://arxiv.org/abs/2406.17557\nThis report introduces FineWeb, a 15-trillion token dataset derived from Common Crawl, and FineWeb-Edu, a 1.3-trillion token educational subset.\nAdam-mini: Use Fewer Learning Rates To Gain More\nby Zhang, Chen, Li,\net al\n. (24 June),\nhttps://arxiv.org/abs/2406.16793\nAdam-mini is a proposed optimizer that achieves comparable or better performance than AdamW while using 45-50% less memory by strategically reducing learning rate resources, partitioning parameters based on Hessian structure, and assigning optimized single learning rates to parameter blocks.\nWARP: On the Benefits of Weight Averaged Rewarded Policies\nby Ramé, Ferret, Vieillard,\net al.\n(24 June),\nhttps://arxiv.org/abs/2406.16768\nThe paper introduces a new alignment strategy for LLMs that merges policies at three stages: using exponential moving average for dynamic KL regularization, spherical interpolation of independently fine-tuned policies, and linear interpolation with initialization.\nSparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers\nby Lou, Jia, Zheng, and Tu (24 June),\nhttps://arxiv.org/abs/2406.16747\nThe authors propose a new sparse attention mechanism for autoregressive Transformers, using a scoring network and differentiable top-k mask operator to select a constant number of KV pairs per query to achieve linear time complexity and a constant memory footprint.\nEfficient Continual Pre-training by Mitigating the Stability Gap\nby Wang, Hu, Xiong,\net al.\n(21 June),\nhttps://arxiv.org/abs/2406.14833\nThis study proposes three strategies to improve continual pretraining of LLMs: multiple epochs on a subset, focusing on high-quality data, and using a mixture similar to pretraining data.\nMoA: Mixture of Sparse Attention for Automatic Large Language Model Compression\nby Fu, Huang, Ning,\net al.\n(21 June),\nhttps://arxiv.org/abs/2406.14909\nMixture of Attention (MoA) automatically optimizes sparse attention patterns for different model components and input lengths in LLMs, improving context length, accuracy, and efficiency over uniform sparse attention approaches.\nLongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs\nby Jiang, Ma, Chen,\net al.\n(21 June),\nhttps://arxiv.org/abs/2406.15319\nLongRAG introduces a new RAG framework using 4K-token retrieval units and long-context LLMs for answer extraction, which improves retrieval performance and achieving state-of-the-art results on question-answering tasks without additional training.\n* A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems\nby Cuconasu, Trappolini, Tonellotto,\net al.\n(21 June),\nhttps://arxiv.org/abs/2406.14972\nThis study challenges conventional wisdom by demonstrating that base LLMs outperform instruction-tuned models in Retrieval Augmented Generation (RAG) tasks.\nCan LLMs Learn by Teaching? A Preliminary Study\nby Ning, Wang, Li, Lin,\net al.\n(20 June),\nhttps://arxiv.org/abs/2406.14629\nThe authors develop and test three methods for implementing \"Learning by Teaching\" in LLMs, mimicking human teaching processes at different levels: observing student feedback, learning from feedback, and iterative learning, to improve model performance without relying on additional human-produced data or stronger models.\n*\nInstruction Pre-Training: Language Models are Supervised Multitask Learners\nby Cheng, Gu, Huang,\net al.\n(20 June),\nhttps://arxiv.org/abs/2406.14491\nThis study introduces a framework for supervised multitask pretraining of LLMs that augments raw corpora with synthetically generated instruction-response pairs.\n* Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?\nby Wu, Zhang, Johnson,\net al.\n(19 June),\nhttps://arxiv.org/abs/2406.13121\nThis study introduces a benchmark for evaluating long-context LLMs on tasks requiring up to millions of tokens, demonstrating that these long-context LLMs can compete with specialized retrieval and RAG systems in in-context retrieval and reasoning tasks.\nJudging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges\nby Ye, Turpin, Li, He,\net al.\n(18 June),\nhttps://arxiv.org/abs/2406.12624\nThis paper evaluates the LLM-as-a-judge paradigm using TriviaQA as a benchmark, comparing 9 judge models and 9 exam taker models against human annotations, revealing that models with high human alignment may not necessarily be the best at ranking exam taker models.\nFrom RAGs to Rich Parameters: Probing How Language Models Utilize External Knowledge Over Parametric Information for Factual Queries\nby Wadhwa, Seetharaman, Aggarwal,\net al.\n(18 June),\nhttps://arxiv.org/abs/2406.12824\nThe authors investigate the mechanics of Retrieval Augmented Generation (RAG) in LLMs to reveal that models predominantly rely on retrieved context information rather than their parametric memory when answering questions, exhibiting a shortcut behavior across different model families.\nSelf-MoE: Towards Compositional Large Language Models with Self-Specialized Experts\nby Kang, Karlinsky, and Luo,\net al.\n(17 June),\nhttps://arxiv.org/abs/2406.12034\nThis paper introduces a method that transforms a monolithic (LLM into a modular system called MiXSE (MiXture of Self-specialized Experts), using self-generated synthetic data to create specialized expert modules with shared base LLM and self-optimized routing.\nMeasuring memorization in RLHF for code completion\nby Pappu, Porter, Shumailov, and Hayes (17 June),\nhttps://arxiv.org/abs/2406.11715\nThis study investigates the impact of Reinforcement Learning with Human Feedback (RLHF) on data memorization in LLMs, focusing on code completion tasks, and finds that while RLHF reduces memorization of data used in reward modeling and reinforcement learning compared to direct finetuning, it largely preserves memorization from the initial finetuning stage.\nHARE: HumAn pRiors, a key to small language model Efficiency\nby Zhang, Jin, Ge,\net al.\n(17 June),\nhttps://arxiv.org/abs/2406.11410\nThis study proposes a principle for leveraging human priors in data construction for Small Language Models (SLMs) that focuses on semantic diversity and data quality consistency while avoiding benchmark data leakage.\nIterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level\nby Kim, Lee, Park,\net al.\n(17 June),\nhttps://arxiv.org/abs/2406.11817\nThis study introduces iterative length-regularized Direct Preference Optimization (iLR-DPO), a method that improves LLM alignment with human preferences while controlling response verbosity.\nUnveiling Encoder-Free Vision-Language Models\nby Choi, Yoon, Lee,\net al\n. (17 June),\nhttps://arxiv.org/abs/2406.11832\nThis study presents an encoder-free vision-language model (VLM) that directly processes both visual and textual inputs in a unified decoder.\n*\nDeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence\nby Zhu, Wang, Lee,\net al.\n(17 June),\nhttps://arxiv.org/abs/2406.11931\nDeepSeek-Coder-V2 is an open-source Mixture-of-Experts code LLM that achieves GPT4-Turbo-level performance on coding tasks through continued pretraining on 6 trillion additional tokens.\nTokenization Falling Short: The Curse of Tokenization\nby Nguyen, Kim, Patel,\net al.\n(17 June),\nhttps://arxiv.org/abs/2406.11687\nThis study investigates the \"curse of tokenization\" in LLMs by examining their performance on complex problem solving, token structure probing, and resilience to typographical variation, revealing that while scaling model size helps, LLMs remain susceptible to tokenization-induced biases\nDataComp-LM: In Search of the Next Generation of Training Sets for Language Models\nby Li, Fang, Smyrnis,\net al.\n(17 June),\nhttps://arxiv.org/abs/2406.11794\nThe authors provide a standardized testbed for experimenting with dataset curation strategies in language model training, including a 240T token corpus, pretraining recipes, and 53 downstream evaluations,\n*\nNemotron-4 340B Technical Report\nby Unknown Authors at NVIDIA (17 June),\nhttps://arxiv.org/abs/2406.11704\nThis technical report accompanies NVIDIA release of the Nemotron-4 340B model family, which performs competitively on various benchmarks and excels in synthetic data generation, along with open-sourcing its data generation pipeline for further research and development.\nmDPO: Conditional Preference Optimization for Multimodal Large Language Models\nby Wang, Zhou, Huang,\net al.\n(17 June),\nhttps://arxiv.org/abs/2406.11839\nmDPO addresses the unconditional preference problem in multimodal DPO by optimizing image preference alongside language preferences and introducing a reward anchor to prevent likelihood decrease for chosen responses.\n*\nHow Do Large Language Models Acquire Factual Knowledge During Pretraining?\nby Chang, Park, Ye,\net al.\n(17 June),\nhttps://arxiv.org/abs/2406.11813\nTask Me Anything\nby Zhang, Huang, Ma,\net al.\n(17 June),\nhttps://arxiv.org/abs/2406.11775\nTask-Me-Anything is a benchmark generation engine that creates tailored benchmarks for multimodal language models by programmatically generating task instances from a vast taxonomy of images and videos.\nTHEANINE: Revisiting Memory Management in Long-term Conversations with Timeline-augmented Response Generation\nby Kim, Ong, Kwon,\net al.\n(16 June),\nhttps://arxiv.org/abs/2406.10996\nTheanine augments LLMs' response generation by using memory timelines (series of memories showing the development and causality of past events) to improve the model's ability to recall and utilize information from lengthy dialogue histories.\nRegularizing Hidden States Enables Learning Generalizable Reward Model for LLMs\nby Yang, Ding, Lin,\net al.\n(14 June)\nhttps://arxiv.org/abs/2406.10216\nThis study proposes enhancing reward model generalization in RLHF by regularizing hidden states through retaining the base model's language model head and incorporating text-generation losses, while simultaneously learning a reward head, thus improving out-of-distribution task performance and mitigating reward over-optimization.\nBe like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs\nby Hans, Wen, Jain,\net al.\n(14 Jun) ,\nhttps://arxiv.org/abs/2406.10209\nThe \"goldfish loss\" technique reduces model memorization in LLMs by randomly excluding a subset of tokens from the loss computation during training, preventing the model from learning complete verbatim sequences from the training data.\nBootstrapping Language Models with DPO Implicit Rewards\nby Chen, Liu, Du,\net al.\n(14 June),\nhttps://arxiv.org/abs/2406.09760\nResearchers find that using the aligned model, an implicit reward model, generated during direct preference optimization (DPO) can itself be used to generate a preference dataset to further substantially improve itself.\nFouRA: Fourier Low Rank Adaptation\nby Borse, Kadambi, Pandey,\net al.\n(13 June),\nhttps://arxiv.org/abs/2406.08798\nThis research introduces FouRA, a new low-rank adaptation (LoRA) method that operates in the Fourier domain and uses adaptive rank selection, addressing issues of data copying and distribution collapse in LoRA-fine-tuned text-to-image diffusion models while improving image quality and generalization.\n*\nAn Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels\nby Nguyen, Mahmoud Assran, Jain,\net al.\n(13 June), https://arxiv.org/abs/2406.09415\nThis research reveals that vanilla Transformers can achieve high performance in various computer vision tasks by treating individual pixels as tokens, which challenges the assumed necessity of locality-based inductive bias in modern vision architectures and suggests new possibilities for future neural network designs in computer vision.\nMLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding\nby Zuhri, Adilazuarda,Purwarianti, and Aji (13 June),\nhttps://arxiv.org/abs/2406.09297\nThis research introduces Multi-Layer Key-Value (MLKV) sharing, a new technique that extends Key-Value (KV) caching across transformer layers, which substantially reduces memory usage during auto-regressive inference beyond existing methods like Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), while maintaining performance on NLP tasks.\nTransformers Meet Neural Algorithmic Reasoners\nby Bounsi, Ibarz, Dudzik,\net al.\n(13 June),\nhttps://arxiv.org/abs/2406.09308\nTransNAR is a hybrid architecture combining Transformers with graph neural network-based neural algorithmic reasoners (NARs), which enables improved performance on algorithmic reasoning tasks by allowing the Transformer to leverage the robust computational capabilities of NARs while maintaining strong natural language understanding.\nDiscovering Preference Optimization Algorithms with and for Large Language Models\nby Lu, Holt, Fanconi,\net al.\n(12 June),\nhttps://arxiv.org/abs/2406.08414\nThe proposed Discovered Preference Optimization method uses LLMs to automatically discover and implement new preference optimization algorithms for improving LLM outputs.\n*\nAn Empirical Study of Mamba-based Language Models\nby Waleffe, Byeon, Riach,\net al.\n(12 June),\nhttps://arxiv.org/abs/2406.07887\nThis research compares 8B-parameter state-space models (Mamba, Mamba-2) and Transformer models trained on large datasets, finding that while pure state-space models match or exceed Transformers on many tasks, they lag behind on tasks requiring strong copying, in-context learning, or long-context reasoning; however, hybrids seem to offer the best of both worlds.\n*\nLarge Language Models Must Be Taught to Know What They Don't Know\nby Kapoor, Gruver, Roberts,\net al.\n(12 June),\nhttps://arxiv.org/abs/2406.08391\nThis research demonstrates that finetuning LLMs on a small dataset of graded examples can produce more reliable uncertainty estimates than prompting alone, with the resulting models capable of estimating uncertainty for themselves and other models.\nLarge Language Model Unlearning via Embedding-Corrupted Prompts\nby Liu, Flannigan, and Liu (12 June),\nhttps://arxiv.org/abs/2406.07933\nThis research introduces embedding-corrupted prompts, a method for selective knowledge unlearning in LLMs that uses prompt classification and embedding corruption to achieve targeted forgetting with minimal side effects across a wide range of model sizes.\nWhat If We Recaption Billions of Web Images with LLaMA-3?\nby Li, Tu, Hui,\net al.\n(12 June)\nhttps://arxiv.org/abs/2406.08478\nThis research demonstrates that using a finetuned Llama 3-powered LLaVA-1.5 multimodal LLM to recaption 1.3 billion images from the DataComp-1B dataset significantly improves the performance of vision-language models in various tasks.\n*\nMagpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing\nby Xu, Jiang, Niu\net al.\n(12 June),\nhttps://arxiv.org/abs/2406.08464\nResearchers propose a synthetic instruction data generation method that generates 300,000 high-quality instruction-response pairs from Llama-3-Instruct; this data can be used for supervised instruction fine-tuning to rival the performance of aligned LLMs without requiring an actual alignment step.\n*\nSamba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling\nby (11 June),\nhttps://arxiv.org/abs/2406.07522\nSamba is a hybrid model combining selective state space models (think Mamba) with sliding window attention that scales efficiently to 3.8B parameters.\n*\nNever Miss A Beat: An Efficient Recipe for Context Window Extension of Large Language Models with Consistent \"Middle\" Enhancement\n(11 June) by Wu, Zhao, and Zheng,\nhttps://arxiv.org/abs/2406.07138\nCREAM is a training-efficient method for extending the context length of LLMs by interpolating positional encodings and using a truncated Gaussian to prioritize middle-context information.\nSimple and Effective Masked Diffusion Language Models\nby Sahoo, Arriola, Schiff,\net al.\n(11 June),\nhttps://arxiv.org/abs/2406.07524\nThis work demonstrates that masked discrete diffusion models, when trained with an effective recipe and a simplified objective, can substantially narrow the performance gap with autoregressive methods in language modeling.\nTextGrad: Automatic \"Differentiation\" via Text\nby Yuksekgonul, Bianchi, Boen,\net al.\n(11 June),\nhttps://arxiv.org/abs/2406.07496\nTextGrad is a framework that leverages LLMs to \"backpropagate\" textual feedback for optimizing building blocks (such as \"tool caller\", \"search engine\", etc.) in compound AI systems.\nAn Image is Worth 32 Tokens for Reconstruction and Generation\nby Yu, Weber, Deng,\net al.\n(11 June),\nhttps://arxiv.org/abs/2406.07550\nThe authors propose a transformer-based 1-dimensional tokenizer for image generation that reduces a 256x256x3 image to just 32 discrete tokens.\n*\nSelf-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teaching\nby Zhang, Peng, Zhou,\net al.\n, (10 June),\nhttps://arxiv.org/abs/2406.06326\nThe Self-Tuning framework improves the knowledge acquisition of LLMs from raw documents through self-teaching tasks focused on memorization, comprehension, and self-reflection.\nTurbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters\nby Song, Xie, Zhang,\net al.\n(10 June),\nhttps://arxiv.org/abs/2406.05955\nThis paper proposes the dReLU activation function and an optimized training data mixture to improve activation sparsity in LLMs.\nHusky: A Unified, Open-Source Language Agent for Multi-Step Reasoning\nby Kim, Paranjape, Khot, and Hajishirzi (10 June),\nhttps://arxiv.org/abs/2406.06469\nHusky is an open-source language agent that learns to reason over a unified action space to tackle diverse tasks involving numerical, tabular, and knowledge-based reasoning by iterating between generating and executing actions with expert models.\nMargin-aware Preference Optimization for Aligning Diffusion Models Without Reference\nby Hong, Paul, Lee,\net al.\n(10 June),\nhttps://arxiv.org/abs/2406.06424\nTo address limitations of traditional alignment techniques like RLHF and DPO, the authors propose Margin-Aware Preference Optimization (MaPO) for text-to-image diffusion models, which maximizes the likelihood margin between preferred and dispreferred image sets without using a reference model.\n*\nAutoregressive Model Beats Diffusion: Llama for Scalable Image Generation\nby Sun, Jian, Chen,\net al.\n(10 June),\nhttps://arxiv.org/abs/2406.06525\nThe authors propose LlamaGen, which involves applying the \"next-token prediction\" paradigm of large language models to image generation.\nCreativity Has Left the Chat: The Price of Debiasing Language Models\nby Mohammidi (8 June),\nhttps://arxiv.org/abs/2406.05587\nThis research reveals that while alignment techniques like RLHF mitigate biases in LLMs, they can diminish the models' creative capabilities, impacting syntactic and semantic diversity, which is crucial for tasks requiring creative output.\n3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination\nby Yang, Chen, Madaan,\net al.\n(7 June),\nhttps://arxiv.org/abs/2406.05132\nThis research introduces 3D-GRAND, a dataset of 40,087 household scenes paired with 6.2 million scene-language instructions, and utilizes instruction tuning and the 3D-POPE benchmark to enhance grounding capabilities and reduce hallucinations in 3D-LLMs\nBERTs are Generative In-Context Learners\nby Samuel (7 June),\nhttps://arxiv.org/abs/2406.04823\nThis paper demonstrates that masked language models, like DeBERTa, can perform in-context learning using a simple inference technique that reformats the sequence of input tokens with mask tokens that resemble the structure of a causal attention mask.\nJune 7, Mixture-of-Agents Enhances Large Language Model Capabilities, https://arxiv.org/abs/2406.04692\nWildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild\nby Lin, Deng, Chandu,\net al.\n(7 June),\nhttps://arxiv.org/abs/2406.04770\nThe authors introduce an automated evaluation framework for benchmarking LLMs using real-world user queries, featuring 1,024 tasks and two advanced metrics, WB-Reward and WB-Score, which provide reliable and interpretable automatic judgments by employing task-specific checklists and structured explanations.\nCRAG -- Comprehensive RAG Benchmark\nby Yang, Sun, Xin,\net al.\n(7 June),\nhttps://arxiv.org/abs/2406.04744\nThis research introduces a factual question answering dataset of 4,409 question-answer pairs with mock APIs simulating web and Knowledge Graph searches, designed to reflect diverse, dynamic real-world QA tasks.\nBoosting Large-scale Parallel Training Efficiency with C4: A Communication-Driven Approach\nby Dong, Luo, Zhang,\net al.\n(7 June),\nhttps://arxiv.org/abs/2406.04594\nThis research introduces C4, a communication-driven solution for parallel training of LLMs, which rapidly identifies and isolates hardware faults and optimizes traffic planning to reduce network congestion, which can cut error-induced overhead by up to 30% and improving runtime performance by up to 15%.\nStep-aware Preference Optimization: Aligning Preference with Denoising Performance at Each Step\nby Liang, Yuan, Gu,\net al.\n(6 June),\nhttps://arxiv.org/abs/2406.04314\nThis research introduces Step-aware Preference Optimization, a post-training approach that independently evaluates and adjusts denoising performance at each step in text-to-image diffusion models, which outperforms Diffusion-DPO in image alignment and aesthetics while offering 20x faster training efficiency.\n*\nAre We Done with MMLU?\nby Gema, Leang, Hong,\net al.\n(6 June),\nhttps://arxiv.org/abs/2406.04127\nThis study identifies numerous errors in the widely-used MMLU benchmark, creates a re-annotated subset called MMLU-Redux that reveals significant discrepancies in reported model performance, and advocates for revising MMLU to improve its reliability.\n* Transformers Need Glasses! Information Over-Squashing in Language Tasks\nby Barbero, Banino, Kapturowski,\net al.\n(6 June),\nhttps://arxiv.org/abs/2406.04267\nThe study analyzes information propagation in LLMs (specifically: decoder-only transformers), revealing a representational collapse phenomenon where distinct input sequences can yield arbitrarily close final token representations, leading to errors in tasks like counting or copying and loss of sensitivity to specific input tokens.\nThe Prompt Report: A Systematic Survey of Prompting Techniques\nby Schulhoff, Ilie, Balepur,\net al.\n(6 June),\nhttps://arxiv.org/abs/2406.06608\nThis 76-page paper aims to provide a clear and organized framework for understanding prompts and prompting techniques.\nBuffer of Thoughts: Thought-Augmented Reasoning with Large Language Models\nby Yang, Yu, Zhang,\net al.\n(6 June),\nhttps://arxiv.org/abs/2406.04271\nThis Buffer of Thoughts approach improves LLMs by retrieving and instantiating thought-templates, which are generic problem-solving blueprints, for reasoning across various domains.\nBlock Transformer: Global-to-Local Language Modeling for Fast Inference\n(4 June) by Ho, Bae, Kim,\net al.\n,\nhttps://arxiv.org/abs/2406.02657\nThe proposed Block Transformer improves inference throughput 10-20x by isolating expensive global attention to lower layers on fixed-size token blocks and applying fast local attention in upper layers.\n*\nScalable MatMul-free Language Modeling\nby Zhu, Zhang, Sifferman,\net al.\n(4 June),\nhttps://arxiv.org/abs/2406.02528\nThis paper presents a scalable MatMul-free language model architecture that replaces matrix multiplications with element-wise products and accumulations using ternary weights that works well even in billion-parameter scales.\nTowards Scalable Automated Alignment of LLMs: A Survey\n, (3 June) by Cao, Lu, Lu,\net al.\nhttps://arxiv.org/abs/2406.01252\nThis paper reviews the recent and emerging automated alignment methods for LLMs that typically follow the instruction finetuning step in an LLM development pipeline.\nThe Geometry of Categorical and Hierarchical Concepts in Large Language Models\nby by Park, Choe, Jiang, and Veitch (3 June),\nhttps://arxiv.org/abs/2406.01506\nUsing a Gemma LLM, this paper extends the linear representation hypothesis, showing that categorical concepts are simplices, hierarchical relations are orthogonal, and complex concepts are polytopes, validated with 957 WordNet concepts.\nOLoRA: Orthonormal Low-Rank Adaptation of Large Language Models\nby Büyükakyüz (3 June),\nhttps://arxiv.org/abs/2406.01775\nOLoRA is an enhancement of Low-Rank Adaptation (LoRA) using orthonormal matrix initialization via QR decomposition, which accelerates the convergence of LLM training compared to regular LoRA.\nSkywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models\nby Wei, Zhu, Zhao\net al.\n(3 June),\nhttps://arxiv.org/abs/2406.06563\nA report that describes some of the approaches and methods behind developing a 146B parameter mixture-of-experts LLM from an existing 13B parameter dense (non-mixture-of-experts) model.\nShow, Don't Tell: Aligning Language Models with Demonstrated Feedback\nby Shaikh, Lam, Hejna,\net al.\n(2 June),\nhttps://arxiv.org/abs/2406.00888\nThe proposed method aligns method aligns LLM outputs to specific user behaviors using fewer than 10 demonstrations as feedback, leveraging imitation learning.\nThis magazine is personal passion project that does not offer direct compensation. However, for those who wish to support me, please consider purchasing a copy of\none of my books\n. If you find them insightful and beneficial, please feel free to recommend them to your friends and colleagues. (Sharing your feedback with others\nvia a book review on Amazon\nhelps a lot, too!)\nBuild A Large Language Model (From Scratch)\n,\nMachine Learning Q And AI\n, and\nMachine Learning with PyTorch and Scikit-Learn\nYour support means a great deal! Thank you!",
  "metadata": {
    "title": "Instruction Pretraining LLMs - by Sebastian Raschka, PhD",
    "description": "The Latest Research in LLM Instruction Finetuning",
    "keywords": null,
    "author": "Sebastian Raschka, PhD",
    "og:type": "article",
    "og:title": "Instruction Pretraining LLMs",
    "og:description": "The Latest Research in Instruction Finetuning",
    "og:image": "https://substackcdn.com/image/fetch/$s_!0QPB!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9aef22-3864-4212-ac39-cf73d569831b_1600x1175.png",
    "og:url": "https://magazine.sebastianraschka.com/p/instruction-pretraining-llms",
    "twitter:title": "Instruction Pretraining LLMs",
    "twitter:description": "The Latest Research in Instruction Finetuning",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!HTAF!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fsebastianraschka.substack.com%2Fapi%2Fv1%2Fpost_preview%2F146761957%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2024-08-12T13:11:01.997Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!0QPB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9aef22-3864-4212-ac39-cf73d569831b_1600x1175.png",
      "bytes": 102434,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-ad79a10b0b0a029b07fd.png",
      "src": "https://substackcdn.com/image/fetch/$s_!0QPB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9aef22-3864-4212-ac39-cf73d569831b_1600x1175.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!akSq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb417eecf-8f0c-4cdb-8a6b-899d515172bd_1510x1526.png",
      "bytes": 277741,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-27531e3a5cc1dd37693c.png",
      "src": "https://substackcdn.com/image/fetch/$s_!akSq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb417eecf-8f0c-4cdb-8a6b-899d515172bd_1510x1526.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!5nwy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8633e941-ac42-4968-a53f-15edd93aad18_1600x1472.png",
      "bytes": 330960,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-5b65a467a3bffad60d3f.png",
      "src": "https://substackcdn.com/image/fetch/$s_!5nwy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8633e941-ac42-4968-a53f-15edd93aad18_1600x1472.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!a2TC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b72fca8-af33-4b89-bb6d-8646b88c51f6_1224x1342.png",
      "bytes": 134326,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-f4b8575fcc665c9166d4.png",
      "src": "https://substackcdn.com/image/fetch/$s_!a2TC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b72fca8-af33-4b89-bb6d-8646b88c51f6_1224x1342.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!ey74!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60d806cb-bbec-4b01-8c8e-7fd52ad28d41_1600x1430.jpeg",
      "bytes": 154517,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-8f03e12811ece28fab6a.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!ey74!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60d806cb-bbec-4b01-8c8e-7fd52ad28d41_1600x1430.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!1qne!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7b54f6a-21ba-4cfc-ba11-84869fe4c1b1_1444x838.png",
      "bytes": 92829,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-81bd256f15b2cb0c3907.png",
      "src": "https://substackcdn.com/image/fetch/$s_!1qne!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7b54f6a-21ba-4cfc-ba11-84869fe4c1b1_1444x838.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!JMNR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F048c0532-67b6-4e47-b9b1-abacbd2d7b33_1600x822.png",
      "bytes": 402214,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-5fffe9211a054194f629.png",
      "src": "https://substackcdn.com/image/fetch/$s_!JMNR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F048c0532-67b6-4e47-b9b1-abacbd2d7b33_1600x822.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!AXY8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92c31ec9-e599-42f9-adbf-7d5af05e9aec_1600x1110.png",
      "bytes": 240407,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-232528a8459d58f7e309.png",
      "src": "https://substackcdn.com/image/fetch/$s_!AXY8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92c31ec9-e599-42f9-adbf-7d5af05e9aec_1600x1110.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!4mSs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e191e7e-4b9c-442d-9fc6-194b8eb0b989_1600x782.png",
      "bytes": 111063,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-3610c7ab9001354cf75e.png",
      "src": "https://substackcdn.com/image/fetch/$s_!4mSs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e191e7e-4b9c-442d-9fc6-194b8eb0b989_1600x782.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!EUJj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57961fbf-d0ce-42d7-bbeb-c7f7222d5221_1600x1016.png",
      "bytes": 80383,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-c2641d0f4a5efb62805c.png",
      "src": "https://substackcdn.com/image/fetch/$s_!EUJj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57961fbf-d0ce-42d7-bbeb-c7f7222d5221_1600x1016.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!vNyr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F469a29ba-6d63-4b4b-8f83-c17359302975_1600x670.png",
      "bytes": 204457,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-37db2452f5d9c4499731.png",
      "src": "https://substackcdn.com/image/fetch/$s_!vNyr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F469a29ba-6d63-4b4b-8f83-c17359302975_1600x670.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!grze!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F336099c3-5356-4b59-bf0e-ab66585fa6aa_1999x1060.png",
      "bytes": 368095,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-9c4a9d7d532e22dee994.png",
      "src": "https://substackcdn.com/image/fetch/$s_!grze!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F336099c3-5356-4b59-bf0e-ab66585fa6aa_1999x1060.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!dSDK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb1b8411-dfa2-4ff9-adac-59b09106df99_962x436.png",
      "bytes": 39952,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-42f686dbf9f31462e8ef.png",
      "src": "https://substackcdn.com/image/fetch/$s_!dSDK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb1b8411-dfa2-4ff9-adac-59b09106df99_962x436.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!o3c-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbdd3b29b-438c-4a08-af54-1c579cdc7463_1600x687.png",
      "bytes": 87816,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-dbec50456627a996b96f.png",
      "src": "https://substackcdn.com/image/fetch/$s_!o3c-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbdd3b29b-438c-4a08-af54-1c579cdc7463_1600x687.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!bxeb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273cd779-88f1-430c-950a-05c75f828d79_1600x760.png",
      "bytes": 90662,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-abc4f51739351362d373.png",
      "src": "https://substackcdn.com/image/fetch/$s_!bxeb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273cd779-88f1-430c-950a-05c75f828d79_1600x760.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!mUlT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47e61f11-d421-4b8d-911c-ef24cff0d15e_1194x744.png",
      "bytes": 89725,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-68a652105c0cc93ed74b.png",
      "src": "https://substackcdn.com/image/fetch/$s_!mUlT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47e61f11-d421-4b8d-911c-ef24cff0d15e_1194x744.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!pi1N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2e16f45-736e-48ef-ac40-c7c418aabb0b_908x598.png",
      "bytes": 74331,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-32758f79f283a42ffbef.png",
      "src": "https://substackcdn.com/image/fetch/$s_!pi1N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2e16f45-736e-48ef-ac40-c7c418aabb0b_908x598.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!nVlV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bbf4f30-91b2-4e5c-988b-be6329f4d1dc_1752x758.png",
      "bytes": 122660,
      "reused": false,
      "local_path": "attachments/ef29388bfc9d/images/body-62ceaa00af0440123ae3.png",
      "src": "https://substackcdn.com/image/fetch/$s_!nVlV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bbf4f30-91b2-4e5c-988b-be6329f4d1dc_1752x758.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://arxiv.org/abs/2406.08464",
      "label": "Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!0QPB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c9aef22-3864-4212-ac39-cf73d569831b_1600x1175.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!akSq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb417eecf-8f0c-4cdb-8a6b-899d515172bd_1510x1526.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/LLMs-from-scratch/tree/main/ch07/05_dataset-generation",
      "label": "Here",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!5nwy!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8633e941-ac42-4968-a53f-15edd93aad18_1600x1472.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/LLMs-from-scratch/blob/main/ch07/05_dataset-generation/instruction-data-llama3-7b.json",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!a2TC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b72fca8-af33-4b89-bb6d-8646b88c51f6_1224x1342.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://mng.bz/M96o",
      "label": "live on the Manning website",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!ey74!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60d806cb-bbec-4b01-8c8e-7fd52ad28d41_1600x1430.jpeg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/LLMs-from-scratch",
      "label": "here on GitHub",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.14491",
      "label": "https://arxiv.org/abs/2406.14491",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!1qne!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7b54f6a-21ba-4cfc-ba11-84869fe4c1b1_1444x838.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2401.14367",
      "label": "https://arxiv.org/abs/2401.14367",
      "type": "a"
    },
    {
      "href": "https://magazine.sebastianraschka.com/i/138555764/mistral-b",
      "label": "https://magazine.sebastianraschka.com/i/138555764/mistral-b",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1809.09600",
      "label": "https://arxiv.org/abs/1809.09600",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!JMNR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F048c0532-67b6-4e47-b9b1-abacbd2d7b33_1600x822.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!AXY8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92c31ec9-e599-42f9-adbf-7d5af05e9aec_1600x1110.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!4mSs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e191e7e-4b9c-442d-9fc6-194b8eb0b989_1600x782.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!EUJj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57961fbf-d0ce-42d7-bbeb-c7f7222d5221_1600x1016.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!vNyr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F469a29ba-6d63-4b4b-8f83-c17359302975_1600x670.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf",
      "label": "Gemma 2 models",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!grze!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F336099c3-5356-4b59-bf0e-ab66585fa6aa_1999x1060.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2310.06825",
      "label": "https://arxiv.org/abs/2310.06825",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!o3c-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbdd3b29b-438c-4a08-af54-1c579cdc7463_1600x687.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.13245",
      "label": "Ainslie",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!bxeb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273cd779-88f1-430c-950a-05c75f828d79_1600x760.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Machine-Learning-AI-Essential-Questions/dp/1718503768/ref=sr_1_1",
      "label": "my Machine Learning Q and AI book",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.16768",
      "label": "WARP: On the Benefits of Weight Averaged Rewarded Policies",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!mUlT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47e61f11-d421-4b8d-911c-ef24cff0d15e_1194x744.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!pi1N!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2e16f45-736e-48ef-ac40-c7c418aabb0b_908x598.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://sebastianraschka.com/books/",
      "label": "my books",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Machine-Learning-AI-Essential-Questions/dp/1718503768",
      "label": "Machine Learning Q and AI",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Machine-Learning-PyTorch-Scikit-Learn-learning/dp/1801819319/ref=sr_1_1?crid=27NRKE8510ZCW&dib=eyJ2IjoiMSJ9.TkNrEFi6U3A1XzQPR1fFxtkV_dPIIK-OXf9buQc9DOIfhEkMakIHB-Vc2_jmSqnevK3jzIMIRVz0VAChRIhNFKXAmxnmCjXHn563FpaCIcpyCvQHafM2tctHm9PSVX11UtMM1pndhNgx9p0K8g7ExA7s-13q5LVOtKoJ-TWB7xswfc-tsL81s1mANhwQmAgDJGJwiewdxKdSEEr1iuoe_8wL9bMLbwxgBl6AuMBLDl0.u59naLiDQN-WVJ7ZGIUS6KEpMOnzd9XpWVMixwa4sjk&dib_tag=se&keywords=raschka+machine+learning&qid=1721327500&sprefix=raschka+machine+learnin%2Caps%2C204&sr=8-1",
      "label": "Machine Learning with PyTorch and Scikit-Learn",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.20094",
      "label": "https://arxiv.org/abs/2406.20094",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2407.00215",
      "label": "https://arxiv.org/abs/2407.00215",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.19774",
      "label": "https://arxiv.org/abs/2406.19774",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.19470",
      "label": "https://arxiv.org/abs/2406.19470",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.19292",
      "label": "https://arxiv.org/abs/2406.19292",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.19395",
      "label": "https://arxiv.org/abs/2406.19395",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.18629",
      "label": "https://arxiv.org/abs/2406.18629",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.18665",
      "label": "https://arxiv.org/abs/2406.18665",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.18219",
      "label": "https://arxiv.org/abs/2406.18219",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.17744",
      "label": "https://arxiv.org/abs/2406.17744",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.17588",
      "label": "https://arxiv.org/abs/2406.17588",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.17557",
      "label": "https://arxiv.org/abs/2406.17557",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.16793",
      "label": "https://arxiv.org/abs/2406.16793",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.16747",
      "label": "https://arxiv.org/abs/2406.16747",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.14833",
      "label": "https://arxiv.org/abs/2406.14833",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.14909",
      "label": "https://arxiv.org/abs/2406.14909",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.15319",
      "label": "https://arxiv.org/abs/2406.15319",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.14972",
      "label": "https://arxiv.org/abs/2406.14972",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.14629",
      "label": "https://arxiv.org/abs/2406.14629",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.13121",
      "label": "https://arxiv.org/abs/2406.13121",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.12624",
      "label": "https://arxiv.org/abs/2406.12624",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.12824",
      "label": "https://arxiv.org/abs/2406.12824",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.12034",
      "label": "https://arxiv.org/abs/2406.12034",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.11715",
      "label": "https://arxiv.org/abs/2406.11715",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.11410",
      "label": "https://arxiv.org/abs/2406.11410",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.11817",
      "label": "https://arxiv.org/abs/2406.11817",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.11832",
      "label": "https://arxiv.org/abs/2406.11832",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.11931",
      "label": "https://arxiv.org/abs/2406.11931",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.11687",
      "label": "https://arxiv.org/abs/2406.11687",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.11794",
      "label": "https://arxiv.org/abs/2406.11794",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.11704",
      "label": "https://arxiv.org/abs/2406.11704",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.11839",
      "label": "https://arxiv.org/abs/2406.11839",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.11813",
      "label": "https://arxiv.org/abs/2406.11813",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.11775",
      "label": "https://arxiv.org/abs/2406.11775",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.10996",
      "label": "https://arxiv.org/abs/2406.10996",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.10216",
      "label": "https://arxiv.org/abs/2406.10216",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.10209",
      "label": "https://arxiv.org/abs/2406.10209",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.09760",
      "label": "https://arxiv.org/abs/2406.09760",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.08798",
      "label": "https://arxiv.org/abs/2406.08798",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.09297",
      "label": "https://arxiv.org/abs/2406.09297",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.09308",
      "label": "https://arxiv.org/abs/2406.09308",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.08414",
      "label": "https://arxiv.org/abs/2406.08414",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.07887",
      "label": "https://arxiv.org/abs/2406.07887",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.08391",
      "label": "https://arxiv.org/abs/2406.08391",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.07933",
      "label": "https://arxiv.org/abs/2406.07933",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.08478",
      "label": "https://arxiv.org/abs/2406.08478",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.07522",
      "label": "https://arxiv.org/abs/2406.07522",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.07138",
      "label": "https://arxiv.org/abs/2406.07138",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.07524",
      "label": "https://arxiv.org/abs/2406.07524",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.07496",
      "label": "https://arxiv.org/abs/2406.07496",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.07550",
      "label": "https://arxiv.org/abs/2406.07550",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.06326",
      "label": "https://arxiv.org/abs/2406.06326",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.05955",
      "label": "https://arxiv.org/abs/2406.05955",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.06469",
      "label": "https://arxiv.org/abs/2406.06469",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.06424",
      "label": "https://arxiv.org/abs/2406.06424",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.06525",
      "label": "https://arxiv.org/abs/2406.06525",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.05587",
      "label": "https://arxiv.org/abs/2406.05587",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.05132",
      "label": "https://arxiv.org/abs/2406.05132",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.04823",
      "label": "https://arxiv.org/abs/2406.04823",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.04770",
      "label": "https://arxiv.org/abs/2406.04770",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.04744",
      "label": "https://arxiv.org/abs/2406.04744",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.04594",
      "label": "https://arxiv.org/abs/2406.04594",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.04314",
      "label": "https://arxiv.org/abs/2406.04314",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.04127",
      "label": "https://arxiv.org/abs/2406.04127",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.04267",
      "label": "https://arxiv.org/abs/2406.04267",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.06608",
      "label": "https://arxiv.org/abs/2406.06608",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.04271",
      "label": "https://arxiv.org/abs/2406.04271",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.02657",
      "label": "https://arxiv.org/abs/2406.02657",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.02528",
      "label": "https://arxiv.org/abs/2406.02528",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.01252",
      "label": "https://arxiv.org/abs/2406.01252",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.01506",
      "label": "https://arxiv.org/abs/2406.01506",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.01775",
      "label": "https://arxiv.org/abs/2406.01775",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.06563",
      "label": "https://arxiv.org/abs/2406.06563",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.00888",
      "label": "https://arxiv.org/abs/2406.00888",
      "type": "a"
    },
    {
      "href": "https://sebastianraschka.com/books",
      "label": "one of my books",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Machine-Learning-AI-Essential-Questions/dp/1718503768/ref=sr_1_1?crid=1566EI5BQC9U0&dib=eyJ2IjoiMSJ9.4oCd5DaBraiVbzZDag-sX4dJQTIguc2mCbDGm1UCKmulcZsTRWmz--_y1AwHt5OmSFglsDpUXQO6FJ_fhs3n9qizrIqlU4STsWxFGor7WdW0QRtPtWgyzz8w0C3PHW8uwsDMJLN4VxjnFIkMizRHBoiHZjLGslLzmiLpzLTTlhbS7bSkhGnUcb1wKkartwWqVtq8c8KbnTdEJ34G6dOlf_YIfsYyG2XOGXneQuNmWh8.AZ3qpJ95F8x9PhDbClERH4iibU3U4r4CTLdtf2r1Nyc&dib_tag=se&keywords=Machine+Learning+Q+and+AI&qid=1717088426&sprefix=%2Caps%2C275&sr=8-1",
      "label": "via a book review on Amazon",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!nVlV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bbf4f30-91b2-4e5c-988b-be6329f4d1dc_1752x758.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "http://mng.bz/M96o",
      "label": "Build A Large Language Model (From Scratch)",
      "type": "a"
    },
    {
      "href": "https://nostarch.com/machine-learning-and-ai-beyond-basics",
      "label": "Machine Learning Q And AI",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Machine-Learning-PyTorch-Scikit-Learn-scikit-learn-ebook-dp-B09NW48MR1/dp/B09NW48MR1/",
      "label": "Machine Learning with PyTorch and Scikit-Learn",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:46:54Z",
  "source_url": "https://magazine.sebastianraschka.com/p/instruction-pretraining-llms",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.08464",
      "bytes": 3951723,
      "reused": false,
      "local_path": "papers/repair/36991b046b784af8c4cb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.14491",
      "bytes": 843992,
      "reused": false,
      "local_path": "papers/repair/8f283a45d06dee001d85.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2401.14367",
      "bytes": 727076,
      "reused": false,
      "local_path": "papers/repair/c5cb7dc5ff01d819b475.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1809.09600",
      "bytes": 2649540,
      "reused": false,
      "local_path": "papers/repair/a0c70d22be73dd81b735.pdf"
    },
    {
      "status": "ok",
      "url": "https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf",
      "bytes": 503119,
      "reused": false,
      "local_path": "papers/repair/e8bf0cc3ae2778bae167.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2310.06825",
      "bytes": 3749788,
      "reused": false,
      "local_path": "papers/repair/0aa8d5bd14ffe7b6d4d5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.13245",
      "bytes": 269116,
      "reused": false,
      "local_path": "papers/repair/5e9c13f12b48cc93f923.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.16768",
      "bytes": 1822203,
      "reused": false,
      "local_path": "papers/repair/b943ddb47a1eb66c3be4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.20094",
      "bytes": 3737124,
      "reused": false,
      "local_path": "papers/repair/28cccf668532b6b78709.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2407.00215",
      "bytes": 1376305,
      "reused": false,
      "local_path": "papers/repair/087ccbb2a3b2515b41e9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.19774",
      "bytes": 1348548,
      "reused": false,
      "local_path": "papers/repair/45b80f3188a54056d4d2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.19470",
      "bytes": 1095491,
      "reused": false,
      "local_path": "papers/repair/4e2dcfc87a4a355d39b9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.19292",
      "bytes": 554857,
      "reused": false,
      "local_path": "papers/repair/b1eb36c782e39b769f6a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.19395",
      "bytes": 983044,
      "reused": false,
      "local_path": "papers/repair/9b1853a347127bd300ce.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.18629",
      "bytes": 1028826,
      "reused": false,
      "local_path": "papers/repair/583fa3adc92f27160231.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.18665",
      "bytes": 809166,
      "reused": false,
      "local_path": "papers/repair/712ffd4203f3f22c8e8f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.18219",
      "bytes": 3164116,
      "reused": false,
      "local_path": "papers/repair/e5df711697af95d21561.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.17744",
      "bytes": 628034,
      "reused": false,
      "local_path": "papers/repair/9f27662ecb2674851779.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.17588",
      "bytes": 893409,
      "reused": false,
      "local_path": "papers/repair/6e78a2c2ac87c8d0344e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.17557",
      "bytes": 3137041,
      "reused": false,
      "local_path": "papers/repair/7774f49e8f8a9b63d955.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.16793",
      "bytes": 13105514,
      "reused": false,
      "local_path": "papers/repair/54c3f3ac6fb5fe2e53da.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.16747",
      "bytes": 2204577,
      "reused": false,
      "local_path": "papers/repair/9d3921113002fefcd037.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.14833",
      "bytes": 1679645,
      "reused": false,
      "local_path": "papers/repair/455620f14fcacb5c848c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.14909",
      "bytes": 1985730,
      "reused": false,
      "local_path": "papers/repair/027f08dc29a41487e4ce.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.15319",
      "bytes": 2322560,
      "reused": false,
      "local_path": "papers/repair/ce48ce1ba0b461070fac.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.14972",
      "bytes": 820135,
      "reused": false,
      "local_path": "papers/repair/63918b776f3563e20d18.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.14629",
      "bytes": 1700963,
      "reused": false,
      "local_path": "papers/repair/fa54535651c14b67276e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.13121",
      "bytes": 2722529,
      "reused": false,
      "local_path": "papers/repair/9dc204a720810b9f3c85.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.12624",
      "bytes": 2159848,
      "reused": false,
      "local_path": "papers/repair/0d72ee0e14e952c8b2e6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.12824",
      "bytes": 1339820,
      "reused": false,
      "local_path": "papers/repair/0b8fca6068eb327ad1e2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.12034",
      "bytes": 1101715,
      "reused": false,
      "local_path": "papers/repair/45708f887a91e96a51b5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.11715",
      "bytes": 1054560,
      "reused": false,
      "local_path": "papers/repair/14e9fc10822326bb0578.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.11410",
      "bytes": 553604,
      "reused": false,
      "local_path": "papers/repair/8740408eb37ec5f509cf.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.11817",
      "bytes": 333877,
      "reused": false,
      "local_path": "papers/repair/55fe06cd948e73dcfbdb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.11832",
      "bytes": 1641808,
      "reused": false,
      "local_path": "papers/repair/a33cc85b4f845dd9ec3d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.11931",
      "bytes": 396316,
      "reused": false,
      "local_path": "papers/repair/32921fc5d72923024f54.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.11687",
      "bytes": 1303224,
      "reused": false,
      "local_path": "papers/repair/4b916e3e9a78c8534236.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.11794",
      "bytes": 1656647,
      "reused": false,
      "local_path": "papers/repair/814551ff9c050c52517f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.11704",
      "bytes": 1370743,
      "reused": false,
      "local_path": "papers/repair/3399068c2dd74b1bc4b5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.11839",
      "bytes": 3110903,
      "reused": false,
      "local_path": "papers/repair/89d2c4b707fc50b252c4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.11813",
      "bytes": 4976449,
      "reused": false,
      "local_path": "papers/repair/33919c8f024594a158ea.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.11775",
      "bytes": 22226097,
      "reused": false,
      "local_path": "papers/repair/01b74c0f0feb48acd4e2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.10996",
      "bytes": 6565493,
      "reused": false,
      "local_path": "papers/repair/f8cd8ac739ae7e31bc7b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.10216",
      "bytes": 917585,
      "reused": false,
      "local_path": "papers/repair/e7cc9c822ba3d9a64b2b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.10209",
      "bytes": 667437,
      "reused": false,
      "local_path": "papers/repair/a3c76a3ff24a477ec8b5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.09760",
      "bytes": 1164588,
      "reused": false,
      "local_path": "papers/repair/b761efe24ebec61d639f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.08798",
      "bytes": 49888641,
      "reused": false,
      "local_path": "papers/repair/83ed063f3277452dd839.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.09297",
      "bytes": 1024948,
      "reused": false,
      "local_path": "papers/repair/ab7e379eb7613c195324.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.09308",
      "bytes": 1228929,
      "reused": false,
      "local_path": "papers/repair/64a7ef11bc8896920ced.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.08414",
      "bytes": 2450912,
      "reused": false,
      "local_path": "papers/repair/c850dbe8d294ffa897d6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.07887",
      "bytes": 903947,
      "reused": false,
      "local_path": "papers/repair/486dc8b5d5386c2f81c1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.08391",
      "bytes": 2800591,
      "reused": false,
      "local_path": "papers/repair/8879a5ead8931b821da0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.07933",
      "bytes": 1227223,
      "reused": false,
      "local_path": "papers/repair/4b3c8d67f56cfba752e7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.08478",
      "bytes": 2681359,
      "reused": false,
      "local_path": "papers/repair/7bc9cb3bb560b9a242ab.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.07522",
      "bytes": 1737433,
      "reused": false,
      "local_path": "papers/repair/50a61fcd4e2ce5e40c01.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.07138",
      "bytes": 1495390,
      "reused": false,
      "local_path": "papers/repair/0669d65802e705c2867e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.07524",
      "bytes": 1034628,
      "reused": false,
      "local_path": "papers/repair/ef0d9a5da3a9065f1034.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.07496",
      "bytes": 1336878,
      "reused": false,
      "local_path": "papers/repair/95c3590eacde2a0e7e21.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.07550",
      "bytes": 6034975,
      "reused": false,
      "local_path": "papers/repair/284efdce3d0f04b98b6e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.06326",
      "bytes": 3488414,
      "reused": false,
      "local_path": "papers/repair/26975298655f5983ef96.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.05955",
      "bytes": 981053,
      "reused": false,
      "local_path": "papers/repair/6501f1f4aab56f7cb06c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.06469",
      "bytes": 2439999,
      "reused": false,
      "local_path": "papers/repair/b517c3379d4174365de2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.06424",
      "bytes": 18217048,
      "reused": false,
      "local_path": "papers/repair/d61873b4ea8471f82786.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.06525",
      "bytes": 21965938,
      "reused": true,
      "local_path": "papers/repair/7947c34b3f7246e1a0ef.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.05587",
      "bytes": 8114934,
      "reused": false,
      "local_path": "papers/repair/b6d79e4ed21f58584733.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.05132",
      "bytes": 10477671,
      "reused": false,
      "local_path": "papers/repair/1369534399c4e8b204a5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.04823",
      "bytes": 657631,
      "reused": false,
      "local_path": "papers/repair/daad7b6857c1f25fca91.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.04770",
      "bytes": 6690142,
      "reused": true,
      "local_path": "papers/repair/ee2914d36268551e8df2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.04744",
      "bytes": 955338,
      "reused": false,
      "local_path": "papers/repair/8d0c0ba6d1b4a290a029.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.04594",
      "bytes": 1715406,
      "reused": false,
      "local_path": "papers/repair/dd9ca412b3ad9ca896bd.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.04314",
      "bytes": 29238678,
      "reused": false,
      "local_path": "papers/repair/338be60980c0d7a24677.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.04127",
      "bytes": 747909,
      "reused": false,
      "local_path": "papers/repair/7c96cd7209c0267434cf.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.04267",
      "bytes": 2624498,
      "reused": false,
      "local_path": "papers/repair/676c71b2dcb6747a136f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.06608",
      "bytes": 3176422,
      "reused": true,
      "local_path": "papers/repair/f2cdbc59d4e9f6c4190f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.04271",
      "bytes": 1658339,
      "reused": false,
      "local_path": "papers/repair/b33cc2ac560f0313d6e3.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.02657",
      "bytes": 1793126,
      "reused": false,
      "local_path": "papers/repair/521baec3b5cb8e16c6ee.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.02528",
      "bytes": 1414451,
      "reused": false,
      "local_path": "papers/repair/b5aede5d2301af3dd561.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.01252",
      "bytes": 2357954,
      "reused": false,
      "local_path": "papers/repair/89d8d8c87baf86d311be.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.01506",
      "bytes": 8373784,
      "reused": false,
      "local_path": "papers/repair/c352104f34d96cdb9d52.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.01775",
      "bytes": 446596,
      "reused": false,
      "local_path": "papers/repair/a7e35f5c8acf7ea18b43.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.06563",
      "bytes": 3048839,
      "reused": false,
      "local_path": "papers/repair/c323f269cd6ebea607d9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.00888",
      "bytes": 1159818,
      "reused": false,
      "local_path": "papers/repair/529e430a9f396b6fd252.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 48517,
  "expected_body_images": 18,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ef29388bfc9d/pages/source.html",
  "local_html": "attachments/ef29388bfc9d/pages/article.html",
  "body_base_path": "attachments/ef29388bfc9d/pages/",
  "body_markdown_path": "attachments/ef29388bfc9d/pages/article.md"
}