{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Sebastian Ruder",
  "title_listed": "Recent Advances in Language Model Fine-tuning",
  "keywords": "微调,语言模型,适配",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.ruder.io/recent-advances-lm-fine-tuning/",
  "publish_date": "2021-02",
  "corpus_stage": "预训练/SFT",
  "notes": "语言模型微调方法综述",
  "id": "2cd93fa63435",
  "collected_at": "2026-10-02T14:14:35",
  "title": "Recent Advances in Language Model Fine-tuning",
  "body_markdown": "language models\nRecent Advances in Language Model Fine-tuning\nThis article provides an overview of recent methods to fine-tune large pre-trained language models.\nSebastian Ruder\n24 Feb 2021\n•\n13 min read\nShare\nFine-tuning a pre-trained language model (LM) has become the de facto standard for doing transfer learning in natural language processing. Over the last three years (\nRuder, 2018\n), fine-tuning (\nHoward & Ruder, 2018\n) has superseded the use of feature extraction of pre-trained embeddings (\nPeters et al., 2018\n) while pre-trained language models are favoured over models trained on translation (\nMcCann et al., 2018\n), natural language inference (\nConneau et al., 2017\n), and other tasks due to their increased sample efficiency and performance (\nZhang and Bowman, 2018\n). The empirical success of these methods has led to the development of ever larger models (\nDevlin et al., 2019\n;\nRaffel et al., 2020\n). Recent models are so large in fact that they can achieve reasonable performance\nwithout\nany parameter updates (\nBrown et al., 2020\n). The limitations of this zero-shot setting (see\nthis section\n), however, make it likely that in order to achieve the best performance or stay reasonably efficient,\nfine-tuning will continue to be the modus operandi when using large pre-trained LMs in practice.\nIn the standard transfer learning setup (see below; see\nthis post\nfor a general overview), a model is first pre-trained on large amounts of unlabelled data using a language modelling loss such as masked language modelling (MLM;\nDevlin et al., 2019\n). The pre-trained model is then fine-tuned on labelled data of a downstream task using a standard cross-entropy loss.\nThe standard pre-training—fine-tuning setting (adapted from\n(Ruder et al., 2019)\n)\nWhile pre-training is compute-intensive, fine-tuning can be done comparatively inexpensively. Fine-tuning is more important for the practical usage of such models as individual pre-trained models are downloaded—and fine-tuned—millions of times (see\nthe Hugging Face models repository\n). Consequently, fine-tuning is the main focus of this post. In particular, I will highlight the most recent advances that have shaped or are likely to change the way we fine-tune language models, which can be seen below.\nOverview of fine-tuning methods discussed in this post.\nAdaptive fine-tuning\nEven though pre-trained language models are more robust in terms of out-of-distribution generalisation than previous models (\nHendrycks et al., 2020\n), they are still poorly equipped to deal with data that is substantially different from the one they have been pre-trained on. Adaptive fine-tuning is a way to bridge such a shift in distribution by fine-tuning the model on data that is closer to the distribution of the target data. Specifically, adaptive fine-tuning involves fine-tuning the model on additional data prior to task-specific fine-tuning, which can be seen below. Importantly, the model is fine-tuned with the pre-training objective, so adaptive fine-tuning only requires unlabelled data.\nAdaptive fine-tuning as part of the standard transfer learning setting. A pre-trained model is trained with the pre-training loss (typically masked language modelling) on data that is closer to the target distribution.\nFormally, given a target domain $\\mathcal{D}_T$ consisting of a feature space $\\mathcal{X}$ and a marginal probability distribution over the feature space $P(X)$ where $X = \\{x_1, \\ldots, x_n \\} \\in \\mathcal{X}$ (\nPan and Yang, 2009\n;\nRuder, 2019\n), adaptive fine-tuning allows us to learn about both the feature space $\\mathcal{X}$ and the distribution of the target data $P(X)$.\nVariants of adaptive fine-tuning—domain, task, and language-adaptive fine-tuning—have been used to adapt a model to data of the target domain, target task, and target language respectively.\nDai and Le (2015)\nfirst showed the benefits of domain-adaptive fine-tuning.\nHoward and Ruder (2018)\nlater demonstrated improved sample efficiency by fine-tuning on in-domain data as part of ULMFiT. They also proposed task-adaptive fine-tuning, which fine-tunes the model with the pre-training objective on the task training data. As the pre-training loss provides richer information for modelling the target data compared to the cross-entropy over one-hot task labels, task-adaptive fine-tuning is useful beyond regular fine-tuning. Alternatively, adaptive and regular fine-tuning can be done jointly via multi-task learning (\nChronopoulou et al., 2019\n).\nDomain and task-adaptive fine-tuning have recently been applied to the latest generation of pre-trained models (\nLogeswaran et al., 2019\n;\nHan and Eisenstein, 2019\n;\nMehri et al., 2019\n).\nGururangan et al. (2020)\nshow that adapting to data of the target domain and target task are complementary. Recently,\nPfeiffer et al. (2020)\nproposed language-adaptive fine-tuning to adapt a model to new languages.\nAn adaptively fine-tuned model is specialised to a particular data distribution, which it will be able to model well. However, this comes at the expense of its ability to be a general model of language.\nAdaptive fine-tuning is thus most useful when high performance on (potentially multiple) tasks of a single domain is important\nand can be computationally inefficient if a pre-trained model should be adapted to a large number of domains.\nBehavioural fine-tuning\nWhile adaptive fine-tuning enables us to specialise our model to $\\mathcal{D}_T$, it does not teach us anything directly about the target task. Formally, a target task $\\mathcal{T}_T$ consists of a label space $\\mathcal{Y}$, a prior distribution $P(Y)$ where $Y = \\{y_1, \\ldots, y_n \\} \\in \\mathcal{Y}$, and a conditional probability distribution $P(Y | X)$. Alternatively, we can teach a model capabilities useful for doing well on the target task by fine-tuning it on relevant tasks, as can be seen below. We will refer to this setting as\nbehavioural fine-tuning\nas it focuses on learning useful behaviours and to distinguish it from adaptive fine-tuning.\nBehavioural fine-tuning of a pre-trained model. The pre-trained model is trained with task-specific supervised or self-supervised objectives on tasks that are relevant for the target task.\nOne way to teach a model relevant capabilities is to fine-tune it on relevant labelled data of a related task prior to task-specific fine-tuning (\nPhang et al., 2018\n). This so-called intermediate-task training works best with tasks that require high-level inference and reasoning capabilities (\nPruksachatkun et al., 2020\n;\nPhang et al., 2020\n). Behavioural fine-tuning with labelled data has been used to teach a model information about named entities (\nBroscheit, 2019)\n, paraphrasing (\nArase and Tsujii, 2019\n), syntax (\nGlavaš and Vulić, 2020\n), answer sentence selection (\nGarg et al., 2020\n), and question answering (\nKhashabi et al., 2020)\n.\nAghajanyan et al. (2021)\nfine-tune on around 50 labelled datasets in a massively multi-task setting and observe that a large, diverse collection of tasks is important for good transfer performance.\nAs supervised data of such high-level reasoning tasks is generally hard to obtain, we can instead train on objectives that teach the model capabilities that are relevant for the downstream task but which can still be learned in a self-supervised manner. For instance,\nDou and Neubig (2021)\nfine-tune a model for word alignment with an objective that teaches it to identify parallel sentences, among others.\nSellam et al. (2020)\nfine-tune BERT for quality evaluation with a range of sentence similarity signals. In both cases, a diversity of learning signals is important.\nAnother effective way is to frame the target task as a form of masked language modelling. To this end,\nBen-David et al. (2020)\nfine-tune a model for sentiment domain adaptation with a pivot-based objective. Others propose pre-training objectives, which can be used similarly during fine-tuning:\nRam et al. (2021)\npre-train a model for QA with a span selection task while\nBansal et al. (2020)\npre-train a model for few-shot learning by automatically generating cloze-style multi-class classification tasks.\nDistinguishing between adaptive and behavioural fine-tuning encourages us to consider the inductive biases we aim to instill in our model and whether they relate to properties of the domain $\\mathcal{D}$ or the task $\\mathcal{T}$\n. Disentangling the role of domain and task is important as information about a domain can often be learned using limited unlabelled data (\nRamponi and Plank, 2020\n) while the acquisition of high-level natural language understanding skills with current methods generally requires billions of pre-training data samples (\nZhang et al., 2020\n).\nThe distinction between task and domain becomes fuzzier, however, when we frame tasks in terms of the pre-training objective. A sufficiently general pre-training task such as MLM may provide useful information for learning $P(Y | X)$ but likely does not contain every signal important for the task. For instance, models pre-trained with MLM struggle with modelling negations, numbers, or named entities (\nRogers et al., 2020\n).\nSimilarly, the use of data augmentation entangles the roles of $\\mathcal{D}$ and $\\mathcal{T}$ as it allows us to encode the desired capabilities directly in the data. For instance, by fine-tuning a model on text where gendered words are replaced with those of the opposite gender, a model can be made more robust to gender bias (\nZhao et al., 2018\n;\nZhao et al., 2019\n;\nManela et al., 2021\n).\nParameter-efficient fine-tuning\nWhen a model needs to be fine-tuned in many settings such as for a large number of users, it is computationally expensive to store a copy of a fine-tuned model for every scenario. Consequently, recent work has focused on keeping most of the model parameters fixed and fine-tuning a small number of parameters per task. In practice, this enables storing a single copy of a large model and many much smaller files with task-specific modifications.\nThe first approaches in this line of work are based on adapters (\nRebuffi et al., 2017\n), small bottleneck layers that are inserted between the layers of a pre-trained model (\nHoulsby et al., 2019\n;\nStickland and Murray, 2019\n) whose parameters are fixed. Adapters render common settings such as storing multiple checkpoints during training as well as more advanced techniques such as checkpoint averaging (\nIzmailov et al., 2018\n), snapshot ensembling (\nHuang et al., 2017\n) and temporal ensembling (\nLaine and Aila, 2017\n) much more space-efficient. Using adapters, a general-purpose model can be efficiently adapted to many settings such as different languages (\nBapna and Firat, 2019\n).\nPfeiffer et al. (2020)\nrecently demonstrated that adapters are modular and can be combined via stacking, which enables learning specialised representations in isolation. This is particularly useful when working with the previously discussed methods:\nan adaptively or behaviourally fine-tuned adapter can be evaluated without any task-specific fine-tuning by stacking a trained task adapter on top of it\n. This setting can be seen below where a task adapter trained on named entity recognition (NER) is stacked on either an English (left) or Quechua language adapter (right).\nTask and language adapters inserted in a Transformer block in the MAD-X framework (\nPfeiffer et al., 2020\n). Adapters learn encapsulated representations and can be replaced with each other, enabling zero-shot transfer.\nWhile adapters modify the model's activations without changing the underlying parameters, another line of work modifies the pre-trained parameters directly. To illustrate this set of methods, we can view fine-tuning as learning how to perturb the parameters of a pre-trained model. Formally, in order to obtain the parameters of a fine-tuned model $\\theta_{\\text{fine-tuned}} \\in \\mathbb{R}^D$ where $D$ is the dimensionality of the model, we learn a task-specific parameter vector $\\theta_{\\text{task}} \\in \\mathbb{R}^D$ that captures how to change the pre-trained model parameters $\\theta_{\\text{pre-trained}} \\in \\mathbb{R}^D$. The fine-tuned parameters are the result of applying the task-specific permutations to the pre-trained parameters:\n\\begin{equation}\n\\theta_{\\text{fine-tuned}} = \\theta_{\\text{pre-trained}} + \\theta_{\\text{task}}\n\\end{equation}\nInstead of storing a copy of $\\theta_{\\text{fine-tuned}}$ for every task, we can store a single copy of $\\theta_{\\text{pre-trained}}$ and a copy of $\\theta_{\\text{task}}$ for every task. This setting is cheaper if we can parameterise $\\theta_{\\text{task}}$ more efficiently. To this end,\nGuo et al. (2020)\nlearn $\\theta_{\\text{task}}$ as a sparse vector.\nAghajanyan et al. (2020)\nset $\\theta_{\\text{task}} = \\theta_\\text{low} \\textbf{M}$ where $\\theta_\\text{low}$ is a low-dimensional vector and $\\textbf{M}$ is a random linear projection (in their case, the FastFood transform (\nLi et al., 2018\n)).\nAlternatively, we can apply modifications only to a subset of the pre-trained parameters. A classic method in computer vision (\nDonahue et al., 2014\n) fine-tunes only the last layer of the model. Let $\\theta_{\\text{pre-trained}}$ be the collection of pre-trained parameters across all $L$ layers of the model, i.e. $\\theta_{\\text{pre-trained}} = \\bigcup\\limits_{l=1}^{L} \\theta_{\\text{pre-trained}}^l$ where $\\theta_{\\text{pre-trained}}^l$ is the parameter vector associated with the $l$-th layer, with analogous notation for $\\theta_{\\text{fine-tuned}}$ and $\\theta_{\\text{task}}$. Fine-tuning only the last layer is then equivalent to:\n\\begin{equation}\n\\begin{split}\n\\theta_{\\text{fine-tuned}} = & (\\bigcup\\limits_{l=1}^{L-1} \\theta_{\\text{pre-trained}}^l) \\\\\n& \\cup (\\theta_{\\text{pre-trained}}^L + \\theta_{\\text{task}}^L)\n\\end{split}\n\\end{equation}\nWhile this works less well in NLP (\nHoward & Ruder, 2018\n), there are other subsets of parameters that are more effective to fine-tune. For instance,\nBen-Zaken et al. (2020)\nachieve competitive performance by only fine-tuning a model's bias parameters.\nAnother line of work prunes parameters of the pre-trained model during fine-tuning. Such methods use different criteria for pruning weights such as based on zero-th or first-order information about a weight's importance (\nSanh et al., 2020\n). As there is limited support of sparse architectures with current hardware, approaches that are\nstructurally\nsparse, i.e. where updates are concentrated in a limited set of layers, matrices, or vectors are currently preferable. For instance, the last few layers of pre-trained models have been shown to be of limited use during fine-tuning and can be randomly reinitialised (\nTamkin et al., 2020\n;\nZhang et al., 2021\n) or even completely removed (\nChung et al., 2021\n).\nWhile pruning methods focus on reducing the total number of parameters of task-specific models, most of the other methods focus on reducing the number of\ntrainable\nparameters—while maintaining a copy of $\\theta_{\\text{pre-trained}}$. The most recent of the latter approaches generally match the performance of full fine-tuning while training around 0.5% of the model's parameters per task (\nPfeiffer et al., 2020\n;\nGuo et al., 2020\n;\nBen-Zaken et al., 2020\n).\nThere is increasing evidence that large pre-trained language models learn representations that compress NLP tasks well (\nLi et al., 2018\n;\nGordon et al., 2020\n;\nAghajanyan et al., 2020\n). This practical evidence coupled with their\nconvenience, availability (\nPfeiffer et al., 2020\n) as well as recent empirical successes make these methods promising both for conducting experiments as well as in practical settings.\nText-to-text fine-tuning\nAnother development in transfer learning is a move from masked language models such as BERT (\nDevlin et al., 2019\n) and RoBERTa (\nLiu et al., 2019\n) to autoregressive models of language such as T5 (\nRaffel et al., 2019\n) and GPT-3 (\nBrown et al., 2020\n). While both sets of methods can be used to assign likelihood scores to text (\nSalazar et al., 2020\n), autoregressive LMs are easier to sample from. In contrast, masked LMs are generally restricted to fill-in-the-blank settings, e.g. (\nPetroni et al., 2019\n).\nThe standard way to use masked LMs for fine-tuning is to replace the output layer used for MLM with a randomly initialised task-specific head that is learned on the target task (\nDevlin et al., 2019\n). Alternatively, the pre-trained model's output layer can be reused by recasting a task as MLM in a cloze-style format (\nTalmor et al., 2020\n;\nSchick and Schütze, 2021\n). Analogously, autoregressive LMs generally cast the target task in a text-to-text format (\nMcCann et al., 2018\n;\nRaffel et al., 2020\n;\nPaolini et al., 2021\n). In both settings,\nthe models are able to benefit from all their pre-trained knowledge and do not need to learn any new parameters from scratch, which improves their sample efficiency.\nIn the extreme when no parameters are fine-tuned, framing a target task in terms of the pre-training objective enables zero-shot or few-shot learning using a task-specific prompt and a small number of examples of a task (\nBrown et al., 2020\n). However, while such few-shot learning is possible, it is not the most effective way to use such models (\nSchick and Schütze, 2020\n; see\nthis post\nfor a brief overview). Learning without updates requires a huge model as the model needs to rely entirely on its existing knowledge. The amount of information available to the model is also restricted by its context window and the prompts shown to the model need to be carefully engineered.\nRetrieval augmentation (see\nthis post\nfor an overview) can be used to off-load the storage of external knowledge and symbolic approaches could be used to teach a model task-specific rules akin to (\nAwasthi et al., 2020\n). Pre-trained models will also become larger and more powerful and may be behaviourally fine-tuned to be good at the zero-shot setting. However, without fine-tuning a model is ultimately limited in its ability to adapt to a new task.\nConsequently,\nfor most practical settings the best path forward arguably is to fine-tune all or a subset of the model's parameters using the methods described in the previous sections.\nIn addition, we will increasingly see an emphasis of pre-trained models' generative capabilities.\nWhile current methods generally focus on modifying a model's natural language input such as via automatic prompt design (\nSchick and Schütze, 2020\n;\nGao et al., 2020\n;\nShin et al., 2020\n), the most effective way to modulate the output of such models will likely act directly on their hidden representations (\nDathathri et al., 2020\n; see\nLillian Weng's post\nfor an overview of methods for controllable generation).\nMitigating fine-tuning instabilities\nA practical problem with fine-tuning pre-trained models is that performance can vary drastically between different runs, particularly on small datasets (\nPhang et al., 2018\n).\nDodge et al., 2020\nfind that both the weight initialisation of the output layer and the order of the training data contribute to variation in performance. As instabilities are generally apparent early in training, they recommend stopping the least promising runs early after 20-30% of training.\nMosbach et al. (2021\n) additionally recommend using small learning rates and to increase the number of epochs when fine-tuning BERT.\nA number of recent methods seek to mitigate instabilities during fine-tuning by relying on adversarial or trust region-based approaches (\nZhu et al., 2019\n;\nJiang et al., 2020\n;\nAghajanyan et al., 2021\n). Such methods generally augment the fine-tuning loss with a regularisation term that bounds the divergence between update steps.\nIn light of the previous section, we can make another recommendation for minimising instabilities during fine-tuning:\nAvoid using a randomly initialised output layer on the target task for small datasets by framing the target task as a form of LM or use behavioural fine-tuning to fine-tune the output layer prior to task-specific fine-tuning.\nWhile text-to-text models are thus more robust to fine-tuning on small datasets, they suffer from instabilities in the few-shot setting and are sensitive to the prompt and few-shot examples (\nZhao et al., 2021\n).\nOverall, as models are increasingly applied to challenging tasks with fewer training examples, it is crucial to develop methods that are robust to possible variations and that can be reliably fine-tuned.\nCitation\nFor attribution in academic contexts, please cite this work as:\n@misc{ruder2021lmfine-tuning,\n  author = {Ruder, Sebastian},\n  title = {{Recent Advances in Language Model Fine-tuning}},\n  year = {2021},\n  howpublished = {\\url{http://ruder.io/recent-advances-lm-fine-tuning}},\n}",
  "metadata": {
    "title": "Recent Advances in Language Model Fine-tuning",
    "description": "This article provides an overview of recent methods to fine-tune large pre-trained language models.",
    "keywords": null,
    "author": null,
    "og:site_name": "ruder.io",
    "og:type": "article",
    "og:title": "Recent Advances in Language Model Fine-tuning",
    "og:description": "This article provides an overview of recent methods to fine-tune large pre-trained language models.",
    "og:url": "https://www.ruder.io/recent-advances-lm-fine-tuning/",
    "og:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2021/02/fine-tuning_methods.png",
    "og:image:width": "1200",
    "og:image:height": "403",
    "twitter:card": "summary_large_image",
    "twitter:title": "Recent Advances in Language Model Fine-tuning",
    "twitter:description": "This post provides an overview of recent methods to fine-tune large pre-trained language models.",
    "twitter:url": "https://www.ruder.io/recent-advances-lm-fine-tuning/",
    "twitter:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2021/02/fine-tuning_methods.png",
    "twitter:label1": "Written by",
    "twitter:data1": "Sebastian Ruder",
    "twitter:label2": "Filed under",
    "twitter:data2": "language models, natural language processing, transfer learning",
    "twitter:site": "@seb_ruder",
    "article:published_time": "2021-02-24T09:00:00.000Z",
    "article:modified_time": "2021-02-24T12:01:05.000Z",
    "article:tag": "transfer learning"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/2cd93fa63435/code/body-000.txt",
      "chars": 213,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2021/02/fine-tuning_methods.png",
      "bytes": 192969,
      "reused": true,
      "local_path": "attachments/2cd93fa63435/images/001.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2021/02/fine-tuning_methods.png",
      "alt": "Recent Advances in Language Model Fine-tuning",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/02/pretraining_finetuning.png",
      "bytes": 155311,
      "reused": true,
      "local_path": "attachments/2cd93fa63435/images/005.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/02/pretraining_finetuning.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/02/fine-tuning_methods_overview.png",
      "bytes": 266885,
      "reused": true,
      "local_path": "attachments/2cd93fa63435/images/006.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/02/fine-tuning_methods_overview.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/02/adaptive_fine-tuning-1.png",
      "bytes": 164402,
      "reused": true,
      "local_path": "attachments/2cd93fa63435/images/007.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/02/adaptive_fine-tuning-1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/02/behavioural_fine-tuning.png",
      "bytes": 168526,
      "reused": true,
      "local_path": "attachments/2cd93fa63435/images/008.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/02/behavioural_fine-tuning.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/02/modular_adapters.png",
      "bytes": 239605,
      "reused": false,
      "local_path": "attachments/2cd93fa63435/images/body-279040a646f4c73bf478.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/02/modular_adapters.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.ruder.io/tag/language-models/",
      "label": "language models",
      "type": "a"
    },
    {
      "href": "https://www.ruder.io/author/sebastian/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://thegradient.pub/nlp-imagenet/?ref=ruder.io",
      "label": "Ruder, 2018",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P18-1031/?ref=ruder.io",
      "label": "Howard & Ruder, 2018",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N18-1202/?ref=ruder.io",
      "label": "Peters et al., 2018",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/2017/file/20c86a628232a67e7bd46f76fba7ce12-Paper.pdf?ref=ruder.io",
      "label": "McCann et al., 2018",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D17-1070/?ref=ruder.io",
      "label": "Conneau et al., 2017",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/W18-5448/?ref=ruder.io",
      "label": "Zhang and Bowman, 2018",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N19-1423/?ref=ruder.io",
      "label": "Devlin et al., 2019",
      "type": "a"
    },
    {
      "href": "https://jmlr.org/papers/v21/20-074.html?ref=ruder.io",
      "label": "Raffel et al., 2020",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf?ref=ruder.io",
      "label": "Brown et al., 2020",
      "type": "a"
    },
    {
      "href": "https://ruder.io/state-of-transfer-learning-in-nlp/?ref=ruder.io",
      "label": "this post",
      "type": "a"
    },
    {
      "href": "https://tiny.cc/NAACLTransfer?ref=ruder.io",
      "label": "(Ruder et al., 2019)",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/models?sort=downloads&ref=ruder.io",
      "label": "the Hugging Face models repository",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.acl-main.244/?ref=ruder.io",
      "label": "Hendrycks et al., 2020",
      "type": "a"
    },
    {
      "href": "https://ieeexplore.ieee.org/document/5288526?ref=ruder.io",
      "label": "Pan and Yang, 2009",
      "type": "a"
    },
    {
      "href": "https://ruder.io/thesis/neural_transfer_learning_for_nlp.pdf?ref=ruder.io",
      "label": "Ruder, 2019",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/2015/file/7137debd45ae4d0ab9aa953017286b20-Paper.pdf?ref=ruder.io",
      "label": "Dai and Le (2015)",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N19-1213/?ref=ruder.io",
      "label": "Chronopoulou et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1335/?ref=ruder.io",
      "label": "Logeswaran et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D19-1433/?ref=ruder.io",
      "label": "Han and Eisenstein, 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1373/?ref=ruder.io",
      "label": "Mehri et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.acl-main.740/?ref=ruder.io",
      "label": "Gururangan et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.emnlp-main.617/?ref=ruder.io",
      "label": "Pfeiffer et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1811.01088?ref=ruder.io",
      "label": "Phang et al., 2018",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.acl-main.467/?ref=ruder.io",
      "label": "Pruksachatkun et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.aacl-main.56/?ref=ruder.io",
      "label": "Phang et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/K19-1063/?ref=ruder.io",
      "label": "Broscheit, 2019)",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D19-1542/?ref=ruder.io",
      "label": "Arase and Tsujii, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2008.06788?ref=ruder.io",
      "label": "Glavaš and Vulić, 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1911.04118?ref=ruder.io",
      "label": "Garg et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.findings-emnlp.171/?ref=ruder.io",
      "label": "Khashabi et al., 2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2101.11038?ref=ruder.io",
      "label": "Aghajanyan et al. (2021)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2101.08231?ref=ruder.io",
      "label": "Dou and Neubig (2021)",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.acl-main.704/?ref=ruder.io",
      "label": "Sellam et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.tacl-1.33/?ref=ruder.io",
      "label": "Ben-David et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2101.00438?ref=ruder.io",
      "label": "Ram et al. (2021)",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.emnlp-main.38/?ref=ruder.io",
      "label": "Bansal et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.coling-main.603/?ref=ruder.io",
      "label": "Ramponi and Plank, 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2011.04946?ref=ruder.io",
      "label": "Zhang et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.tacl-1.54/?ref=ruder.io",
      "label": "Rogers et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N18-2003/?ref=ruder.io",
      "label": "Zhao et al., 2018",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N19-1064/?ref=ruder.io",
      "label": "Zhao et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2101.09688?ref=ruder.io",
      "label": "Manela et al., 2021",
      "type": "a"
    },
    {
      "href": "https://proceedings.neurips.cc/paper/2017/hash/e7b24b112a44fdd9ee93bdf998c6ca0e-Abstract.html?ref=ruder.io",
      "label": "Rebuffi et al., 2017",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v97/houlsby19a.html?ref=ruder.io",
      "label": "Houlsby et al., 2019",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v97/stickland19a.html?ref=ruder.io",
      "label": "Stickland and Murray, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1803.05407?ref=ruder.io",
      "label": "Izmailov et al., 2018",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=BJYwwY9ll&ref=ruder.io",
      "label": "Huang et al., 2017",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=BJ6oOfqge&ref=ruder.io",
      "label": "Laine and Aila, 2017",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D19-1165/?ref=ruder.io",
      "label": "Bapna and Firat, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2012.07463?ref=ruder.io",
      "label": "Guo et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2012.13255?ref=ruder.io",
      "label": "Aghajanyan et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=ryup8-WCW&ref=ruder.io",
      "label": "Li et al., 2018",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v32/donahue14.html?ref=ruder.io",
      "label": "Donahue et al., 2014",
      "type": "a"
    },
    {
      "href": "https://nlp.biu.ac.il/~yogo/bitfit.pdf?ref=ruder.io",
      "label": "Ben-Zaken et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/2020/file/eae15aabaa768ae4a5993a8a4f4fa6e4-Paper.pdf?ref=ruder.io",
      "label": "Sanh et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.findings-emnlp.125/?ref=ruder.io",
      "label": "Tamkin et al., 2020",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=cO1IH43yUF&ref=ruder.io",
      "label": "Zhang et al., 2021",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=xpFFI_NtgpW&ref=ruder.io",
      "label": "Chung et al., 2021",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.repl4nlp-1.18/?ref=ruder.io",
      "label": "Gordon et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.emnlp-demos.7/?ref=ruder.io",
      "label": "Pfeiffer et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1907.11692?ref=ruder.io",
      "label": "Liu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.acl-main.240/?ref=ruder.io",
      "label": "Salazar et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D19-1250/?ref=ruder.io",
      "label": "Petroni et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.tacl-1.48/?ref=ruder.io",
      "label": "Talmor et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2001.07676?ref=ruder.io",
      "label": "Schick and Schütze, 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1806.08730?ref=ruder.io",
      "label": "McCann et al., 2018",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=US-TP-xnXI&ref=ruder.io",
      "label": "Paolini et al., 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.07118?ref=ruder.io",
      "label": "Schick and Schütze, 2020",
      "type": "a"
    },
    {
      "href": "https://ruder.io/research-highlights-2020/?ref=ruder.io",
      "label": "this post",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=SkeuexBtDr&ref=ruder.io",
      "label": "Awasthi et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2012.15723?ref=ruder.io",
      "label": "Gao et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.15980?ref=ruder.io",
      "label": "Shin et al., 2020",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=H1edEyBKDS&ref=ruder.io",
      "label": "Dathathri et al., 2020",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/lil-log/2021/01/02/controllable-neural-text-generation.html?ref=ruder.io",
      "label": "Lillian Weng's post",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2002.06305?ref=ruder.io",
      "label": "Dodge et al., 2020",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=nzpLWnVAyah&ref=ruder.io",
      "label": "Mosbach et al. (2021",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=BygzbyHFvB&ref=ruder.io",
      "label": "Zhu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.acl-main.197/?ref=ruder.io",
      "label": "Jiang et al., 2020",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=OQ08SN70M1V&ref=ruder.io",
      "label": "Aghajanyan et al., 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2102.09690?ref=ruder.io",
      "label": "Zhao et al., 2021",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:54Z",
  "source_url": "https://www.ruder.io/recent-advances-lm-fine-tuning/",
  "papers": [
    {
      "status": "ok",
      "url": "https://papers.nips.cc/paper/2017/file/20c86a628232a67e7bd46f76fba7ce12-Paper.pdf?ref=ruder.io",
      "bytes": 412761,
      "reused": false,
      "local_path": "papers/repair/49fa7f8c50c5ba3b65e7.pdf"
    },
    {
      "status": "ok",
      "url": "https://papers.nips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf?ref=ruder.io",
      "bytes": 1283694,
      "reused": false,
      "local_path": "papers/repair/ef081e90efc54a8bdee8.pdf"
    },
    {
      "status": "http_404",
      "url": "https://ruder.io/thesis/neural_transfer_learning_for_nlp.pdf?ref=ruder.io",
      "retried_at": "2026-10-20T17:50:39Z"
    },
    {
      "status": "ok",
      "url": "https://papers.nips.cc/paper/2015/file/7137debd45ae4d0ab9aa953017286b20-Paper.pdf?ref=ruder.io",
      "bytes": 102897,
      "reused": false,
      "local_path": "papers/repair/d779ec60ed6329db92af.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1811.01088",
      "bytes": 323286,
      "reused": false,
      "local_path": "papers/repair/900b122b4c467abf7740.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2008.06788",
      "bytes": 1343111,
      "reused": false,
      "local_path": "papers/repair/ff196ca13d8a36733007.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1911.04118",
      "bytes": 2090120,
      "reused": false,
      "local_path": "papers/repair/db2b28e9c161d58c0f84.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2101.11038",
      "bytes": 341656,
      "reused": false,
      "local_path": "papers/repair/7c695d308c524af8f395.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2101.08231",
      "bytes": 2861196,
      "reused": false,
      "local_path": "papers/repair/860de36317f5fb279e2c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2101.00438",
      "bytes": 1383346,
      "reused": false,
      "local_path": "papers/repair/852d501bb634fc467507.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2011.04946",
      "bytes": 1475419,
      "reused": false,
      "local_path": "papers/repair/da42fd2f7d423e08fda5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2101.09688",
      "bytes": 344139,
      "reused": false,
      "local_path": "papers/repair/5691d706e6494ee307ac.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1803.05407",
      "bytes": 733206,
      "reused": true,
      "local_path": "papers/repair/27c3305367428ea24842.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2012.07463",
      "bytes": 513545,
      "reused": false,
      "local_path": "papers/repair/d63745904b0f32bb43c6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2012.13255",
      "bytes": 383666,
      "reused": false,
      "local_path": "papers/repair/6bfe07a7b46d9da8811c.pdf"
    },
    {
      "status": "ok",
      "url": "https://nlp.biu.ac.il/~yogo/bitfit.pdf?ref=ruder.io",
      "bytes": 187901,
      "reused": false,
      "local_path": "papers/repair/14c2cd6cd14cd2f14157.pdf"
    },
    {
      "status": "ok",
      "url": "https://papers.nips.cc/paper/2020/file/eae15aabaa768ae4a5993a8a4f4fa6e4-Paper.pdf?ref=ruder.io",
      "bytes": 2040321,
      "reused": false,
      "local_path": "papers/repair/356203d70bc75b162977.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1907.11692",
      "bytes": 209675,
      "reused": true,
      "local_path": "papers/repair/8acd467c11db68744d3f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2001.07676",
      "bytes": 486241,
      "reused": true,
      "local_path": "papers/repair/b0a84807f9d9b85ecc2d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1806.08730",
      "bytes": 1187133,
      "reused": false,
      "local_path": "papers/repair/e2a98f1a611cef094a63.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.07118",
      "bytes": 468372,
      "reused": false,
      "local_path": "papers/repair/e950ca6026acc3b4cc56.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2012.15723",
      "bytes": 1441484,
      "reused": false,
      "local_path": "papers/repair/f9ff70b06a4d3ed4bae9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.15980",
      "bytes": 663180,
      "reused": true,
      "local_path": "papers/repair/5fc241cafafb7eeca9db.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2002.06305",
      "bytes": 843093,
      "reused": false,
      "local_path": "papers/repair/1d5cf4282687ee0760e6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2102.09690",
      "bytes": 695668,
      "reused": true,
      "local_path": "papers/repair/f904b24f8b1404d53b54.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 20752,
  "expected_body_images": 6,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "http_404",
  "raw_html": "attachments/2cd93fa63435/pages/source.html",
  "local_html": "attachments/2cd93fa63435/pages/article.html",
  "body_base_path": "attachments/2cd93fa63435/pages/",
  "body_markdown_path": "attachments/2cd93fa63435/pages/article.md"
}