{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Sebastian Ruder",
  "title_listed": "An Overview of Instruction Tuning Data",
  "keywords": "指令微调,数据,LLM",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.ruder.io/an-overview-of-instruction-tuning-data/",
  "publish_date": "2023-11",
  "corpus_stage": "数据处理",
  "notes": "指令微调数据集综述",
  "id": "06f0d8b26bb4",
  "collected_at": "2026-10-02T14:15:14",
  "title": "An Overview of Instruction Tuning Data",
  "body_markdown": "An Overview of Instruction Tuning Data\nThis post covers a range of widely used instruction tuning datasets, as well as important characteristics of instruction tuning data and best practices for using the datasets.\nSebastian Ruder\n15 Nov 2023\n•\n9 min read\nShare\nTimeline of a subset of public instruction tuning datasets (\nLongpre et al., 2023\n).\nNLP and ML have gone through several phases of how models are trained in recent years. With the arrival of pre-trained models such as BERT, fine-tuning pre-trained models for downstream tasks became the norm. The increasing capabilities of ever larger models then enabled in-context learning via prompting. Recently, instruction tuning has become the newest method to make LLMs useful.\nThis post covers a range of widely used instruction tuning datasets, as well as important characteristics of instruction tuning data and best practices for using the datasets.\nThis post was first published in NLP News in\ntwo\narticles\n.\nWhat is Instruction Tuning?\nThe main difference between instruction tuning and standard supervised fine-tuning lies in the data that the model is trained on. Whereas supervised fine-tuning trains models on\ninput examples and their corresponding outputs\n, instruction tuning\naugments input-output examples with instructions\n, which enables instruction-tuned models to generalize more easily to new tasks.\nPretrain-finetuning vs prompting vs instruction tuning (\nWei et al., 2022\n).\nMethods differ based on how the instruction tuning data is constructed.\nZhang et al. (2023)\nprovide a good overview of existing instruction datasets. Existing datasets fall roughly into two main categories: a) instructions are added to existing NLP tasks; and b) data from (a) is used to condition a model to generate new instruction-input-output tuples.\nImportant Data Characteristics\nThere are a several aspects to consider when using instruction tuning datasets:\nData source:\nHow was the data obtained? Most datasets have been generated using ChatGPT. They may thus inherit biases of the source model or may be noisy. Human-written examples are more expensive to obtain but are more high quality.\nData quality:\nWas any filtering done to improve the quality of the generated data? In most cases, filtering is based on simple heuristics or a pre-trained model, which can result in noisy data. For instance, the authors of\nOpenAssistant Conversations\nwent the extra mile and obtained human-annotated data quality labels.\nDomain and language coverage:\nMost datasets cover general QA-style use cases and are in English. However, similar methods can be used to obtain data in other domains or languages.\nNumber of dialog turns:\nA\ndialog turn\nis an utterance by one speaker. Most datasets are single-turn, i.e., they consist of a prompt and a single response. Multi-turn data may be necessary to train a more conversational model.\nLicense terms:\nData generated using OpenAI models is subject to the\nOpenAI terms of use\n, which prohibit using the data to develop competing models. So look for data with a more permissive license to avoid any legal complications.\nInstruction Tuning Datasets\nLet us now look at a representative set of widely used instruction tuning datasets, in order of their release:\nNatural Instructions\n(\nMishra et al., 2022\n)\n: 193k instruction-output examples sourced from 61 existing English NLP tasks. Crowd-sourcing instructions from each dataset are aligned to a common schema.\nInstructions are thus more structured\ncompared to other datasets.\nOutputs are relatively short\n, however, which makes the data less useful for generating long-form content.\nTwo instances of Natural Instructions. The instruction schema covers multiple fields including a definition, things to avoid, and positive and negative examples.\nNatural Instructions v2 / Super-Natural Instructions (\nWang et al., 2022\n)\n: A crowd-sourced collection of instruction data based on existing NLP tasks and simple synthetic tasks. It includes 5M examples across 76 tasks in 55 languages. Compared to Natural Instructions,\ninstructions are simplified\n; they consist of a task definition and positive and negative examples with explanations.\nUnnatural Instructions (\nHonovich et al., 2023\n)\n: An automatically collected instruction dataset of 240k examples where InstructGPT (text-davinci-002) is prompted with three Super-Natural Instructions examples—consisting of an instruction, input, possible output constraints—and asked to generate a new example. The output is generated separately by conditioning on the generated instruction, input, and constraints. The generated instructions are then further paraphrased by prompting the model. Unnatural Instructions covers a\nmore diverse set of tasks\nthan Super-Natural Instructions; while many examples reflect classical NLP tasks, it also includes examples of other interesting tasks.\nExamples of eight generated instructions that differ from classical NLP tasks in Unnatural Instructions.\nSelf-Instruct\n(Wang et al., 2023)\n: Similar to Unnatural Instructions, Self-Instruct consists of 82k examples automatically generated using InstructGPT conditioned on a set of\nseed task examples\n(175 tasks in total; one example per task; 8 examples are sampled for in-context learning). Self-Instruct decouples the example generation by first generating the instruction, then the input (conditioned on instruction), and then the output. For classification tasks, the authors first generate the possible output labels and then condition the input generation on each class label to avoid biasing towards a specific label. While the generated instructions are mostly valid, the\ngenerated outputs are often noisy\n.\nExamples of generated instruction-input-output tuples in Self-Instruct.\nP3\n(\nPublic Pool of Prompts; Sanh et al., 2022\n)\n: A crowd-sourced collection of prompts for 177 English NLP tasks. For each dataset, about 11 different prompts are available on average, which enables studying the impact of different prompt formulations. Compared to the instructions in the above instruction datasets,\nP3 prompts are often shorter and less elaborate\n.\nP3 prompt templates for two existing NLP tasks. Prompt templates use fields of the raw data (e.g.,\n{Document}\n) and template metadata (e.g.,\n{Choices[label]}\n).\nxP3, xP3mt\n(Muennighoff et al., 2023)\n: An extension of P3 including 19 multilingual datasets and 11 code datasets, with English prompts. They also release a machine-translated version of the data (xP3mt), which contains prompts automatically translated into 20 languages. Fine-tuning on multilingual tasks with English prompts further improves performance beyond only fine-tuning on English instruction data.\nPrompt templates in P3, xP3, and xP3mt. Prompts are in English in P3 and xP3.\nFlan 2021 / Muffin (\nWei et al., 2022\n):\nPrompts for 62 English text datasets, with 10 prompt templates for each task. For classification tasks, an\nOPTIONS\nsuffix is appended to the input in order to indicate output constraints.\nInstructions, inputs, and outputs for three tasks in Flan 2021.\nFlan 2022 (\nChung et al., 2022\n)\n: A combination of Flan 2021, P3, Super-Natural Instructions, and additional reasoning, dialog, and program synthesis datasets. The 9 new reasoning datasets are additionally annotated with chain-of-thought (CoT;\nWei et al., 2022\n) annotations.\nFlan 2022 instructions enable fine-tuning with and without exemplars (few-shot vs zero-shot) and with and without chain-of-thought.\nOpt-IML Bench (\nIyer et al., 2022\n)\n: A combination of Super-Natural Instructions, P3, and Flan 2021. They additionally include dataset collections on cross-task transfer, knowledge grounding, dialogue, and a larger number of chain-of-thought datasets.\nDifferent prompt formulations of the COPA task in Opt-IML Bench.\nAlpaca data\n(Taori et al., March 2023)\n: 52k English instruction examples generated using OpenAI’s text-davinci-003 with self-instruct. The authors applied\nsome modifications\nto simplify the data generation pipeline and lower costs—the final data cost less than $500 to generate!\nEvol-Instruct\n(Xu et al., April 2023)\n: A rewritten set of 250k English instruction-response pairs based on the Alpaca data. Instructions are rewritten a) to make them more complex or b) to create a new, more specialized instruction by prompting ChatGPT. In a second step, ChatGPT is used to generate the corresponding responses. Low-quality instruction-response pairs are filtered using heuristics. This process is repeated three times.\nVicuna ShareGPT data\n(Chiang et al., March 2023)\n: 70k English conversations shared by users and scraped from\nsharegpt.com\n. Pre-processing involved converting HTML to markdown, filtering out low-quality samples, and splitting lengthy conversations into smaller segments. Compared to the above single-turn datasets, the ShareGPT conversations often consist of multiple turns and are thus more useful for training a model to leverage the context of the conversation. The conversations may be owned by the users so their use is potentially problematic.\nBaize data\n(Xu et al., April 2023)\n:\n54k and 57k English multi-turn dialog examples (3.4 turns on average) generated with ChatGPT using questions from Quora and StackOverflow datasets respectively as seeds. ChatGPT simulates both the human and AI participants of the conversation.\n1\nIn addition, they also generated 47k dialogs in the medical domain based on\nMedQuAD\nquestions.\nMulti-turn dialogue data generated using ChatGPT based on a seed from Quora\n(Xu et al., 2023)\n.\ndatabricks-dolly-15k\n(\nConover et al., April 2023\n):\n15k English instruction-following examples written by Databricks employees. Crucially, both instructions and answers are human-generated. This in contrast to the other datasets above where instruction and/or answers are generated by ChatGPT. Examples cover 7 use cases: open QA, closed QA, information extraction and summarization of Wikipedia data, brainstorming, classification, and creative writing. Compared to the other datasets above, the data is released under a permissive license that also allows for commercial use.\nAn Open QA example in\ndatabricks-dolly-15k\n.\nOpenAssistant Conversations\n(\nKöpf et al., April 2023\n):\n11k crowd-sourced multilingual instruction-following conversations (for 52k examples, only the prompts are available). Human annotators generated messages for both the assistant and the human participant. The data differs in several aspects from the other datasets:\n1)\nit is multilingual (42.8% examples are in English, 31.4% in Spanish, and the rest in other languages);\n2)\nannotators annotated the quality of prompts and responses (460k quality ratings overall); and\n3)\nthe annotators were provided with detailed guidelines, both for writing prompts and for acting as the assistant. The data uses a permissive license, which allows commercial use.\nLIMA data\n(Zhou et al., May 2023)\n:\n1k training and 300 test answer–response pairs mostly sampled from StackExchange, wikiHow and the Pushshift Reddit dataset with around 400 written by the paper authors. A nice observation of this study is that\ntraining on this small set of curated instruction data outperforms training on the much larger, noisier Alpaca data\n.\nExamples from the manually authored portion of the LIMA dataset.\nInstruction Tuning Best Practices\nLongpre et al. (2023)\nand\nIyer et al. (2022)\nablate several important aspects of using instruction tuning data:\nMixing few-shot settings.\nTraining with mixed zero-shot and few-shot prompts significantly improves performance in both settings.\nTask diversity.\nLarge models benefit from continuously increasing the number of tasks.\nData augmentation.\nAugmenting the data such as by inverting inputs and outputs (e.g., turning a question answering task into a question generation task) is beneficial.\nMixing weights.\nWhen using a combination of instruction tuning datasets, appropriately tuning the mixing weights is important.\nTakeaways\n✅ Quality > quantity.\nAs\nZhou et al. (2023)\nobserve, training on a small set of high-quality data outperforms instruction-tuning on larger, noisier data. Using more diverse prompts and quality filtering both improve performance.\n🧑‍🎓\nImitation != mastery.\nModels that are instruction-tuned on ChatGPT-generated data mimic ChatGPT’s style (and may thus fool human raters!) but not its factuality (\nGudibande et al., May 2023\n). They perform worse on standard benchmarks. Using stronger base models is the best way to address this.\n🏛️\nThe stronger the base, the better.\nMore powerful base models also produce stronger instruction-tuned models\n(Wang et al., June 2023)\n.\n🥇\nCombination wins.\nCombining multiple instruction-tuning datasets results in the best average performance across tasks\n(Wang et al., June 2023)\n. Dataset mixing and developing\nmodular instruction-tuned models\nare thus important research directions.\nFuture Directions\nUnderstanding instruction-tuning.\nWhile we have seen a proliferation of instruction-tuning datasets, we still lack a clear understanding of what makes a good instruction and good instruction–response pairs. There is much anecdotal knowledge when it comes to creating good model prompts—but to my knowledge it is unclear how instruction–following data can be created at scale in a more principled manner.\nImproving data quality.\nTo improve model performance, we need to develop more reliable methods to identify high-quality examples and filter out undesirable ones. In a similar vein, it is important to develop methods that allow us to identify how a particular instance affects model behavior and alignment at test time.\nEvaluating instruction-tuned models.\nIn light of the biases of both human and automatic evaluations, there is no clear gold standard for how to evaluate instruction-tuned models. Evaluating a model on a set of tests that can be efficiently and automatically evaluated is one way to side-step this issue, see LMentry (\nEfrat et al., ACL 2023\n), M2C (\nHlavnova & Ruder, ACL 2023\n), IFEval (\nZhou et al., Nov 2023\n), etc but these are restricted to a certain set of use cases. In general, it is crucial to design evaluations with a target application in mind.",
  "metadata": {
    "title": "An Overview of Instruction Tuning Data",
    "description": "This post covers a range of widely used instruction tuning datasets, as well as important characteristics of instruction tuning data and best practices for using the datasets.",
    "keywords": null,
    "author": null,
    "og:site_name": "ruder.io",
    "og:type": "article",
    "og:title": "An Overview of Instruction Tuning Data",
    "og:description": "This post covers a range of widely used instruction tuning datasets, as well as important characteristics of instruction tuning data and best practices for using the datasets.",
    "og:url": "https://www.ruder.io/an-overview-of-instruction-tuning-data/",
    "og:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2024/10/instruction_tuning_data-1-1.jpg",
    "og:image:width": "1200",
    "og:image:height": "272",
    "twitter:card": "summary_large_image",
    "twitter:title": "An Overview of Instruction Tuning Data",
    "twitter:description": "This post covers a range of widely used instruction tuning datasets, as well as important characteristics of instruction tuning data and best practices for using the datasets.",
    "twitter:url": "https://www.ruder.io/an-overview-of-instruction-tuning-data/",
    "twitter:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2024/10/instruction_tuning_data-1-1.jpg",
    "twitter:label1": "Written by",
    "twitter:data1": "Sebastian Ruder",
    "twitter:site": "@seb_ruder",
    "article:published_time": "2023-11-15T20:03:00.000Z",
    "article:modified_time": "2024-10-23T13:20:49.000Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2024/10/instruction_tuning_data-1-1.jpg",
      "bytes": 54161,
      "reused": true,
      "local_path": "attachments/06f0d8b26bb4/images/001.jpg",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2024/10/instruction_tuning_data-1-1.jpg",
      "alt": "An Overview of Instruction Tuning Data",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/instruct_tuning_methods.png",
      "bytes": 111079,
      "reused": true,
      "local_path": "attachments/06f0d8b26bb4/images/005.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/instruct_tuning_methods.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/natural_instructions.jpg",
      "bytes": 205807,
      "reused": true,
      "local_path": "attachments/06f0d8b26bb4/images/008.jpg",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/natural_instructions.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/unnatural_instructions.jpg",
      "bytes": 211700,
      "reused": true,
      "local_path": "attachments/06f0d8b26bb4/images/011.jpg",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/unnatural_instructions.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/self-instruct.jpg",
      "bytes": 90538,
      "reused": true,
      "local_path": "attachments/06f0d8b26bb4/images/014.jpg",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/self-instruct.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/p3.png",
      "bytes": 98224,
      "reused": true,
      "local_path": "attachments/06f0d8b26bb4/images/017.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/p3.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/xp3.png",
      "bytes": 84086,
      "reused": true,
      "local_path": "attachments/06f0d8b26bb4/images/020.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/xp3.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/flan.png",
      "bytes": 120234,
      "reused": true,
      "local_path": "attachments/06f0d8b26bb4/images/023.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/flan.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/flan_2022.png",
      "bytes": 176741,
      "reused": true,
      "local_path": "attachments/06f0d8b26bb4/images/026.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/flan_2022.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/opt_iml_bench.jpg",
      "bytes": 122449,
      "reused": true,
      "local_path": "attachments/06f0d8b26bb4/images/029.jpg",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/opt_iml_bench.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/baize.png",
      "bytes": 181160,
      "reused": true,
      "local_path": "attachments/06f0d8b26bb4/images/032.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/baize.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/databricks.png",
      "bytes": 283761,
      "reused": true,
      "local_path": "attachments/06f0d8b26bb4/images/036.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/databricks.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/lima.jpg",
      "bytes": 273911,
      "reused": false,
      "local_path": "attachments/06f0d8b26bb4/images/body-75725ed13c857a9d3764.jpg",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/lima.jpg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.ruder.io/author/sebastian/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2301.13688",
      "label": "Longpre et al., 2023",
      "type": "a"
    },
    {
      "href": "https://newsletter.ruder.io/p/instruction-tuning-vol-1?ref=ruder.io",
      "label": "two",
      "type": "a"
    },
    {
      "href": "https://newsletter.ruder.io/p/instruction-tuning-vol-2?ref=ruder.io",
      "label": "articles",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=gEZrGCozdqR&ref=ruder.io",
      "label": "Wei et al., 2022",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2308.10792?ref=ruder.io",
      "label": "Zhang et al. (2023)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2304.07327?ref=ruder.io",
      "label": "OpenAssistant Conversations",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Turn-taking?ref=ruder.io",
      "label": "dialog turn",
      "type": "a"
    },
    {
      "href": "https://openai.com/policies/terms-of-use?ref=ruder.io",
      "label": "OpenAI terms of use",
      "type": "a"
    },
    {
      "href": "https://instructions.apps.allenai.org/?ref=ruder.io",
      "label": "Natural Instructions",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2022.acl-long.244/?ref=ruder.io",
      "label": "Mishra et al., 2022",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2022.emnlp-main.340/?ref=ruder.io",
      "label": "Wang et al., 2022",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d8e6cae-c3f0-4776-a355-a7d9b149b6bc_718x929.png?ref=ruder.io",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2023.acl-long.806/?ref=ruder.io",
      "label": "Honovich et al., 2023",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2023.acl-long.754/?ref=ruder.io",
      "label": "(Wang et al., 2023)",
      "type": "a"
    },
    {
      "href": "https://github.com/yizhongw/self-instruct/blob/main/human_eval/user_oriented_instructions.jsonl?ref=ruder.io",
      "label": "seed task examples",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/bigscience/P3?ref=ruder.io",
      "label": "P3",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=9Vrb9D0WI4&ref=ruder.io",
      "label": "Public Pool of Prompts; Sanh et al., 2022",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F995dcc96-3ae0-4c56-9a09-0c7d642b0822_1339x451.png?ref=ruder.io",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/bigscience-workshop/xmtf?ref=ruder.io",
      "label": "xP3, xP3mt",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2023.acl-long.891/?ref=ruder.io",
      "label": "(Muennighoff et al., 2023)",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2e8b350-d214-405f-9848-fca7e9a8b244_1470x403.png?ref=ruder.io",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fceb4e28d-424d-450d-a29e-598d883d1fb2_1229x500.png?ref=ruder.io",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2210.11416?ref=ruder.io",
      "label": "Chung et al., 2022",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2301.13688?ref=ruder.io",
      "label": ")",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=_VjQlMeSB_J&ref=ruder.io",
      "label": "Wei et al., 2022",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2212.12017?ref=ruder.io",
      "label": "Iyer et al., 2022",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3456a516-cb35-421f-8423-bebe296da46d_1399x597.png?ref=ruder.io",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/tatsu-lab/stanford_alpaca?ref=ruder.io",
      "label": "Alpaca data",
      "type": "a"
    },
    {
      "href": "https://crfm.stanford.edu/2023/03/13/alpaca.html?ref=ruder.io",
      "label": "(Taori et al., March 2023)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2304.12244?ref=ruder.io",
      "label": "(Xu et al., April 2023)",
      "type": "a"
    },
    {
      "href": "https://lmsys.org/blog/2023-03-30-vicuna/?ref=ruder.io",
      "label": "(Chiang et al., March 2023)",
      "type": "a"
    },
    {
      "href": "https://sharegpt.com/?ref=ruder.io",
      "label": "sharegpt.com",
      "type": "a"
    },
    {
      "href": "https://github.com/project-baize/baize-chatbot?ref=ruder.io",
      "label": "Baize data",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2304.01196?ref=ruder.io",
      "label": "(Xu et al., April 2023)",
      "type": "a"
    },
    {
      "href": "https://github.com/abachaa/MedQuAD?ref=ruder.io",
      "label": "MedQuAD",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/databricks/databricks-dolly-15k?ref=ruder.io",
      "label": "databricks-dolly-15k",
      "type": "a"
    },
    {
      "href": "https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm?ref=ruder.io",
      "label": "Conover et al., April 2023",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/OpenAssistant/oasst1?ref=ruder.io",
      "label": "OpenAssistant Conversations",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.11206?ref=ruder.io",
      "label": "(Zhou et al., May 2023)",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F64293fa7-f3f0-4cd6-80e5-01cdad4493b4_1597x980.png?ref=ruder.io",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.15717?ref=ruder.io",
      "label": "Gudibande et al., May 2023",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2306.04751?ref=ruder.io",
      "label": "(Wang et al., June 2023)",
      "type": "a"
    },
    {
      "href": "https://www.modulardeeplearning.com/?ref=ruder.io",
      "label": "modular instruction-tuned models",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2023.findings-acl.666/?ref=ruder.io",
      "label": "Efrat et al., ACL 2023",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2023.acl-long.396/?ref=ruder.io",
      "label": "Hlavnova & Ruder, ACL 2023",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2311.07911?ref=ruder.io",
      "label": "Zhou et al., Nov 2023",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F524c44f6-f75e-4f74-a66c-bc80f1318d8a_1322x803.png?ref=ruder.io",
      "label": "",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:55Z",
  "source_url": "https://www.ruder.io/an-overview-of-instruction-tuning-data/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2301.13688",
      "bytes": 899298,
      "reused": false,
      "local_path": "papers/repair/5f6eb2980131e507aabc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2308.10792",
      "bytes": 6604498,
      "reused": false,
      "local_path": "papers/repair/4074ca73e0a64973d59b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2304.07327",
      "bytes": 2199088,
      "reused": false,
      "local_path": "papers/repair/1c750677799860ea18c4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2210.11416",
      "bytes": 1557309,
      "reused": false,
      "local_path": "papers/repair/face74216dc244661745.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2301.13688",
      "bytes": 899298,
      "reused": true,
      "local_path": "papers/repair/5f6eb2980131e507aabc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2212.12017",
      "bytes": 1077867,
      "reused": false,
      "local_path": "papers/repair/aeb2c246ace451b30d1b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2304.12244",
      "bytes": 1199147,
      "reused": false,
      "local_path": "papers/repair/cd796aa2f5ce0be534bc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2304.01196",
      "bytes": 440687,
      "reused": false,
      "local_path": "papers/repair/280aeda9bf49d40f4cbe.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.11206",
      "bytes": 499719,
      "reused": false,
      "local_path": "papers/repair/946d6ba40f3dc00cc689.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.15717",
      "bytes": 1051380,
      "reused": true,
      "local_path": "papers/repair/bc4305bd40664b25adcc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2306.04751",
      "bytes": 1583165,
      "reused": false,
      "local_path": "papers/repair/ed65efcfae6acad8a8fb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2311.07911",
      "bytes": 464152,
      "reused": true,
      "local_path": "papers/repair/aa7d0c8c51f0f0ea8fdd.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 14176,
  "expected_body_images": 13,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/06f0d8b26bb4/pages/source.html",
  "local_html": "attachments/06f0d8b26bb4/pages/article.html",
  "body_base_path": "attachments/06f0d8b26bb4/pages/",
  "body_markdown_path": "attachments/06f0d8b26bb4/pages/article.md"
}