{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Sebastian Ruder",
  "title_listed": "The State of Transfer Learning in NLP",
  "keywords": "迁移学习,NLP,预训练",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.ruder.io/state-of-transfer-learning-in-nlp/",
  "publish_date": "2019-08",
  "corpus_stage": "预训练/SFT",
  "notes": "NLP迁移学习现状经典综述（NAACL教程）",
  "id": "dba5b30674d7",
  "collected_at": "2026-10-02T14:14:35",
  "title": "The State of Transfer Learning in NLP",
  "body_markdown": "transfer learning\nThe State of Transfer Learning in NLP\nThis post expands on the NAACL 2019 tutorial on Transfer Learning in NLP. It highlights key insights and takeaways and provides updates based on recent work.\nSebastian Ruder\n18 Aug 2019\n•\n15 min read\nShare\nUpdate 16.10.2020: Added\nChinese\nand\nSpanish\ntranslations.\nThis post expands on the\nNAACL 2019 tutorial on Transfer Learning in NLP\n. The tutorial was organized by Matthew Peters, Swabha Swayamdipta, Thomas Wolf, and me. In this post, I highlight key insights and takeaways and provide updates based on recent work. You can see the structure of this post below:\nThe\nslides\n, a\nColaboratory notebook\n, and\ncode\nof the tutorial are available online.\nIntroduction\nFor an overview of what transfer learning is, have a look at\nthis blog post\n. Our go-to definition throughout this post will be the following, which is illustrated in the diagram below:\nTransfer learning is a means to extract knowledge from a source setting and apply it to a different target setting.\nAn illustration of the process of transfer learning.\nIn the span of little more than a year, transfer learning in the form of pretrained language models has\nbecome ubiquitous in NLP\nand has contributed to the state of the art on a wide range of tasks. However, transfer learning is not a recent phenomenon in NLP. One illustrative example is progress on the task of Named Entity Recognition (NER), which can be seen below.\nPerformance on Named Entity Recognition (NER) on CoNLL-2003 (English) over time.\nThroughout its history, most of the major improvements on this task have been driven by different forms of transfer learning: from early self-supervised learning with auxiliary tasks (\nAndo and Zhang, 2005\n) and phrase & word clusters (\nLin and Wu, 2009\n) to the language model embeddings (\nPeters et al., 2017\n) and pretrained language models (\nPeters et al., 2018\n;\nAkbik et al., 2018\n;\nBaevski et al., 2019\n) of recent years.\nThere are different types of transfer learning common in current NLP. These can be roughly classified along three dimensions based on a) whether the source and target settings deal with the same task; and b) the nature of the source and target domains; and c) the order in which the tasks are learned. A taxonomy that highlights the variations can be seen below:\nA taxonomy for transfer learning in NLP (\nRuder, 2019\n).\nSequential transfer learning is the form that has led to the biggest improvements so far. The general practice is to pretrain representations on a large unlabelled text corpus using your method of choice and then to adapt these representations to a supervised target task using labelled data as can be seen below.\nThe general procedure of sequential transfer learning.\nMajor themes\nSeveral major themes can be observed in how this paradigm has been applied:\nFrom words to words-in-context\nOver time, representations incorporate more context. Early approaches such as word2vec (\nMikolov et al., 2013\n) learned a single representation for every word independent of its context. Later approaches then scaled these representations to sentences and documents (\nLe and Mikolov, 2014\n;\nConneau et al., 2017\n). Current approaches learn word representations that change based on the word's context (\nMcCann et al., 2017\n;\nPeters et al., 2018\n).\nLM pretraining\nMany successful pretraining approaches are based on variants of language modelling (LM). Advantages of LM are that it does not require any human annotation and that many languages have enough text available to learn reasonable models. In addition, LM is versatile and enables learning both sentence and word representations with a variety of objective functions.\nFrom shallow to deep\nOver the last years, state-of-the-art models in NLP have become progressively deeper. Up to two years ago, the state of the art on most tasks was a 2-3 layer deep BiLSTM, with machine translation being an outlier with 16 layers (\nWu et al., 2016\n). In contrast, current models like BERT-Large and GPT-2 consist of 24 Transformer blocks and recent models are even deeper.\nPretraining vs target task\nThe choice of pretraining and target tasks is closely intertwined. For instance, sentence representations are not useful for word-level predictions, while span-based pretraining is important for span-level predictions. On the whole, for the best target performance, it is beneficial to choose a similar pretraining task.\nPretraining\nWhy does language modelling work so well?\nThe remarkable success of pretrained language models is surprising. One reason for the success of language modelling may be that it is a very difficult task, even for humans. To have any chance at solving this task, a model is required to learn about syntax, semantics, as well as certain facts about the world. Given enough data, a large number of parameters, and enough compute, a model can do a reasonable job. Empirically, language modelling works better than other pretraining tasks such as translation or autoencoding (\nZhang et al. 2018\n;\nWang et al., 2019\n).\nA recent predictive-rate distortion (PRD) analysis of human language (\nHahn and Futrell, 2019\n) suggests that human language—and language modelling—has infinite statistical complexity but that it can be approximated well at lower levels. This observation has two implications: 1) We can obtain good results with comparatively small models; and 2) there is a lot of potential for scaling up our models. For both implications we have empirical evidence, as we can see in the next sections.\nSample efficiency\nOne of the main benefits of pretraining is that it reduces the need for annotated data. In practice, transfer learning has often been shown to achieve similar performance compared to a non-pretrained model with 10x fewer examples or more as can be seen below for ULMFiT (\nHoward and Ruder, 2018\n).\nPerformance of a model trained from scratch (blue) vs. two pretrained models fine-tuned on labelled target data (orange) as well as unlabelled target data (green) respectively (\nHoward and Ruder, 2018\n).\nScaling up pretraining\nPretrained representations can generally be improved by jointly increasing the number of model parameters and the amount of pretraining data. Returns start to diminish as the amount of pretraining data grows huge. Current performance curves such as the one below, however, do not indicate that we have reached a plateau. We can thus expect to see even bigger models trained on more data.\nAverage GLUE score with different amounts of Common Crawl data for pretraining (\nBaevski et al., 2019\n).\nRecent examples of this trend are\nERNIE 2.0\n,\nXLNet\n,\nGPT-2 8B\n, and\nRoBERTa\n. The latter in particular finds that simply training BERT for longer and on more data improves results, while GPT-2 8B reduces perplexity on a language modelling dataset (though only by a comparatively small factor).\nCross-lingual pretraining\nA major promise of pretraining is that it can help us bridge the digital language divide and can enable us learn NLP models for more of the world's 6,000 languages. Much work on cross-lingual learning has focused on training separate word embeddings in different languages and learning to align them (\nRuder et al., 2019\n). In the same vein, we can learn to align contextual representations (\nSchuster et al., 2019\n). Another common method is to share a subword vocabulary and train one model on many languages (\nDevlin et al., 2019\n;\nArtetxe and Schwenk, 2019\n;\nMulcaire et al., 2019\n;\nLample and Conneau, 2019\n). While this is easy to implement and is a strong cross-lingual baseline, it leads to under-representation of low-resource languages (\nHeinzerling and Strube, 2019\n). Multilingual BERT in particular has been the subject of much recent attention (\nPires et al., 2019\n;\nWu and Dredze, 2019\n). Despite its strong zero-shot performance, dedicated monolingual language models often are competitive, while being more efficient (\nEisenschlos et al., 2019\n).\nPractical considerations\nPretraining is cost-intensive. Pretraining the Transformer-XL style model we used in the tutorial takes 5h–20h on 8 V100 GPUs (a few days with 1 V100) to reach a good perplexity. Sharing pretrained models is thus very important. Pretraining is relatively robust to the choice of hyper-parameters—apart from needing a learning rate warm-up for transformers. As a general rule, your model should not have enough capacity to overfit if your dataset is large enough. Masked language modeling (as in BERT) is typically 2-4 times slower to train than standard LM as masking only a fraction of words yields a smaller signal.\nWhat is in a representation?\nRepresentations have been shown to be predictive of certain linguistic phenomena such as alignments in translation or syntactic hierarchies. Better performance has been achieved when pretraining with syntax; even when syntax is not explicitly encoded, representations still learn some notion of syntax (\nWilliams et al. 2018\n). Recent work has furthermore shown that knowledge of syntax can be distilled efficiently into state-of-the-art models (\nKuncoro et al., 2019\n). Network architectures generally determine what is in a representation. For instance, BERT has been observed to capture syntax (\nTenney et al., 2019\n;\nGoldberg, 2019\n). Different architectures show different layer-wise trends in terms of what information they capture (\nLiu et al., 2019\n).\nThe general setup in probing tasks used to study linguistic knowledge within contextual word representations (\nLiu et al., 2019\n).\nThe information that a model captures also depends how you look at it: Visualizing activations or attention weights provides a bird's eye view of the model's knowledge, but focuses on a few samples; probes that train a classifier on top of learned representations in order to predict certain properties (as can be seen above) discover corpus-wide specific characteristics, but may introduce their own biases; finally, network ablations are great for improving the model, but may be task-specific.\nAdaptation\nFor adapting a pretrained model to a target task, there are several orthogonal directions we can make decisions on: architectural modifications, optimization schemes, and whether to obtain more signal.\nArchitectural modifications\nFor architectural modifications, the two general options we have are:\na) Keep the pretrained model internals unchanged\nThis can be as simple as adding one or more linear layers on top of a pretrained model, which is commonly done with BERT. Instead, we can also use the model output as input to a separate model, which is often beneficial when a target task requires interactions that are not available in the pretrained embedding, such as span representations or modelling cross-sentence relations.\nb) Modify the pretrained model internal architecture\nOne reason why we might want to do this is in order to adapt to a structurally different target task such as one with several input sequences. In this case, we can use the pretrained model to initialize as much as possible of a structurally different target task model. We might also want to apply task-specific modifications such as adding skip or residual connections or attention. Finally, modifying the target task parameters may reduce the number of parameters that need to be fine-tuned by adding bottleneck modules (“adapters”) between the layers of the pretrained model (\nHoulsby et al., 2019\n;\nStickland and Murray, 2019\n).\nAn adapter layer (right) as used in a Transformer block (left) (\nHoulsby et al., 2019\n).\nOptimization schemes\nIn terms of optimizing the model, we can choose which weights we should update and how and when to update those weights.\nWhich weights to update\nFor updating the weights, we can either tune or not tune (the pretrained weights):\na) Do not change the pretrained weights (feature extraction)\nIn practice, a linear classifier is trained on top of the pretrained representations. The best performance is typically achieved by using the representation not just of the top layer, but learning a linear combination of layer representations (\nPeters et al., 2018\n,\nRuder et al., 2019\n). Alternatively, pretrained representations can be used as features in a downstream model. When adding adapters, only the adapter layers are trained.\nUse of a pretrained model as features in a separate downstream model.\nb) Change the pretrained weights (fine-tuning)\nThe pretrained weights are used as initialization for parameters of the downstream model. The whole pretrained architecture is then trained during the adaptation phase.\nHow and when to update the weights\nThe main motivation for choosing the order and how to update the weights is that we want to avoid overwriting useful pretrained information and maximize positive transfer. Related to this is the concept of catastrophic forgetting (\nMcCloskey & Cohen, 1989\n;\nFrench, 1999\n), which occurs if a model forgets the task it was originally trained on. In most settings, we only care about the performance on the target task, but this may differ depending on the application.\nA guiding principle for updating the parameters of our model is to update them progressively from top-to-bottom in time, in intensity, or compared to a pretrained model:\na) Progressively in time (freezing)\nThe main intuition is that training all layers at the same time on data of a different distribution and task may lead to instability and poor solutions. Instead, we train layers individually to give them time to adapt to the new task and data. This goes back to layer-wise training of early deep neural networks (\nHinton et al., 2006\n;\nBengio et al., 2007\n). Recent approaches (\nFelbo et al., 2017\n;\nHoward and Ruder, 2018\n;\nChronopoulou et al., 2019\n) mostly vary in the combinations of layers that are trained together; all train all parameters jointly in the end. Unfreezing has not been investigated in detail for Transformer models.\nb) Progressively in intensity (lower learning rates)\nWe want to use lower learning rates to avoid overwriting useful information. Lower learning rates are particularly important in lower layers (as they capture more general information), early in training (as the model still needs to adapt to the target distribution), and late in training (when the model is close to convergence). To this end, we can use discriminative fine-tuning (\nHoward and Ruder, 2018\n), which decays the learning rate for each layer as can be seen below. In order to maintain lower learning rates early in training, a triangular learning rate schedule can be used, which is also known as learning rate warm-up in Transformers.\nLiu et al. (2019)\nrecently suggest that warm-up reduces variance in the early stage of training.\nDiscriminative fine-tuning (\nHoward and Ruder, 2018\n).\nc) Progressively vs. a pretrained model (regularization)\nOne way to minimize catastrophic forgetting is to encourage target model parameters to stay close to the parameters of the pretrained model using a regularization term (\nWiese et al., CoNLL 2017\n,\nKirkpatrick et al., PNAS 2017\n).\nTrade-offs and practical considerations\nIn general, the more parameters you need to train from scratch the slower your training will be. Feature extraction requires adding more parameters than fine-tuning (\nPeters et al., 2019\n), so is typically slower to train. Feature extraction, however, is more space-efficient when a model needs to be adapted to many tasks as it only requires storing one copy of the pretrained model in memory. Adapters strike a balance by adding a small number of additional parameters per task.\nIn terms of performance, no adaptation method is clearly superior in every setting. If source and target tasks are dissimilar, feature extraction seems to be preferable (\nPeters et al., 2019\n). Otherwise, feature extraction and fine-tuning often perform similar, though this depends on the budget available for hyper-parameter tuning (fine-tuning may often require a more extensive hyper-parameter search). Anecdotally, Transformers are easier to fine-tune (less sensitive to hyper-parameters) than LSTMs and may achieve better performance with fine-tuning.\nHowever, large pretrained models (e.g. BERT-Large) are prone to degenerate performance when fine-tuned on tasks with small training sets. In practice, the observed behavior is often “on-off”: the model either works very well or does not work at all as can be seen in the figure below. Understanding the conditions and causes of this behavior is an open research question.\nDistribution of task scores across 20 random restarts for BERT (red) and BERT that was fine-tuned on MNLI (green) when fine-tuning on no more than 5k examples for each task (\nPhang et al., 2018\n).\nGetting more signal\nThe target task is often a low-resource task. We can often improve the performance of transfer learning by combining a diverse set of signals:\nSequential adaptation\nIf related tasks are available, we can fine-tune our model first on a related task with more data before fine-tuning it on the target task. This\nhelps particularly for tasks with limited data and similar tasks (\nPhang et al., 2018\n) and improves sample efficiency on the target task (\nYogatama et al., 2019\n).\nMulti-task fine-tuning\nAlternatively, we can also fine-tune the model jointly on related tasks together with the target task. The related task can also be an unsupervised auxiliary task. Language modelling is a good choice for this and has been shown to help even without pretraining (\nRei et al., 2017\n). The task ratio can optionally be annealed to de-emphasize the auxiliary task towards the end of training (\nChronopoulou et al., NAACL 2019\n). Language model fine-tuning is used as a separate step in ULMFiT (\nHoward and Ruder, 2018\n). Recently, multi-task fine-tuning has led to improvements even with many target tasks (\nLiu et al., 2019\n,\nWang et al., 2019\n).\nDataset slicing\nRather than fine-tuning with auxiliary tasks, we can use\nauxiliary heads that are trained only on particular subsets of the data\n. To this end, we would first analyze the errors of the model, use heuristics to automatically identify challenging subsets of the training data, and then train auxiliary heads jointly with main head.\nSemi-supervised learning\nWe can also use semi-supervised learning methods to make our model's predictions more consistent by perturbing unlabelled examples. The perturbation can be noise, masking (\nClark et al., 2018\n), or data augmentation, e.g. back-translation (\nXie et al., 2019\n).\nEnsembling\nTo improve performance the predictions of models fine-tuned with different hyper-parameters, fine-tuned with different pretrained models, or trained on different target tasks or dataset splits may be combined.\nDistilling\nFinally, large models or ensembles of models may be distilled into a single, smaller model. The model can also be a lot simpler (Tang et al., 2019) or have a different inductive bias (\nKuncoro et al., 2019\n). Multi-task fine-tuning can also be combined with distillation (\nClark et al., 2019\n).\nDown-stream applications\nPretraining large-scale models is costly, not only in terms of computation but also in terms of the environmental impact (\nStrubell et al., 2019\n). Whenever possible, it's best to use open-source models. If you need to train your own models, please share your pretrained models with the community.\nFrameworks and libraries\nFor sharing and accessing pretrained models, different options are available:\nHubs\nHubs are central repositories that provide a common API for accessing pretrained models. The two most common hubs are\nTensorFlow Hub\nand\nPyTorch Hub\n. Hubs are generally simple to use; however, they act more like a black-box as the source code of the model cannot be easily accessed. In addition, modifying the internals of a pretrained model architecture can be difficult.\nAuthor released checkpoints\nCheckpoint files generally contain all the weights of a pretrained model. In contrast to hub modules, the model graph still needs to be created and model weights need to be loaded separately. As such, checkpoint files are more difficult to use than hub modules, but provide you with full control over the model internals.\nThird-party libraries\nSome third-party libraries like\nAllenNLP\n,\nfast.ai\n, and\npytorch-transformers\nprovide easy access to pretrained models. Such libraries typically enable fast experimentation and cover many standard use cases for transfer learning.\nFor examples of how such models and libraries can be used for downstream tasks, have a look at the\ncode snippets\nin the slides, the\nColaboratory notebook\n, and the\ncode\n.\nOpen problems and future directions\nThere are many open problems and interesting future research directions. Below is just an updated selection. For more pointers, have a look at\nthe slides\n.\nShortcomings of pretrained language models\nPretrained language models are still bad at fine-grained linguistic tasks (\nLiu et al., 2019\n), hierarchical syntactic reasoning (\nKuncoro et al., 2019\n), and common sense (when you actually make it difficult;\nZellers et al., 2019\n). They still fail at natural language generation, in particular maintaining long-term dependencies, relations, and coherence. They also tend to overfit to surface form information when fine-tuned and can still mostly be seen as ‘rapid surface learners’.\nAs we have noted above, particularly large models that are fine-tuned on small amounts of data are difficult to optimize and suffer from high variance. Current pretrained language models are also very large. Distillation and pruning are two ways to deal with this.\nPretraining tasks\nWhile the language modelling objective has shown to be effective empirically, it has its weaknesses. Lately, we have seen that bidirectional context and modelling contiguous word sequences is particularly important. Maybe most importantly, language modelling encourages a focus on syntax and word co-occurrences and only provides a weak signal for capturing semantics and long-term context. We can take inspiration from other forms of self-supervision. In addition, we can design specialized pretraining tasks that explicitly learn certain relationships (\nJoshi et al., 2019\n,\nSun et al., 2019\n).\nOn the whole, it is difficult to learn certain types of information from raw text. Recent approaches incorporate structured knowledge (\nZhang et al., 2019\n;\nLogan IV et al., 2019\n) or leverage multiple modalities (\nSun et al., 2019\n;\nLu et al., 2019\n) as two potential ways to mitigate this problem.\nCitation\nIf you found this post helpful, consider citing\nthe tutorial\nas:\n@inproceedings{ruder2019transfer,\n  title={Transfer Learning in Natural Language Processing},\n  author={Ruder, Sebastian and Peters, Matthew E and Swayamdipta, Swabha and Wolf, Thomas},\n  booktitle={Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials},\n  pages={15--18},\n  year={2019}\n}\nTranslations\nThis article has been translated into the following languages:\nChinese\n(by Sambodhi)\nSpanish\n(by Ibidem Group)",
  "metadata": {
    "title": "The State of Transfer Learning in NLP",
    "description": "This post expands on the NAACL 2019 tutorial on Transfer Learning in NLP. It highlights key insights and takeaways and provides updates based on recent work.",
    "keywords": null,
    "author": null,
    "og:site_name": "ruder.io",
    "og:type": "article",
    "og:title": "The State of Transfer Learning in NLP",
    "og:description": "This post expands on the NAACL 2019 tutorial on Transfer Learning in NLP. It highlights key insights and takeaways and provides updates based on recent work.",
    "og:url": "https://www.ruder.io/state-of-transfer-learning-in-nlp/",
    "og:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2019/08/transfer_learning_methods_small.png",
    "og:image:width": "1200",
    "og:image:height": "464",
    "twitter:card": "summary_large_image",
    "twitter:title": "The State of Transfer Learning in NLP",
    "twitter:description": "This post expands on the NAACL 2019 tutorial on Transfer Learning in NLP. It highlights key insights and takeaways and provides updates based on recent work.",
    "twitter:url": "https://www.ruder.io/state-of-transfer-learning-in-nlp/",
    "twitter:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2019/08/transfer_learning_methods_small.png",
    "twitter:label1": "Written by",
    "twitter:data1": "Sebastian Ruder",
    "twitter:label2": "Filed under",
    "twitter:data2": "transfer learning, natural language processing, events",
    "twitter:site": "@seb_ruder",
    "article:published_time": "2019-08-18T15:22:00.000Z",
    "article:modified_time": "2020-10-16T21:47:19.000Z",
    "article:tag": "events"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/dba5b30674d7/code/body-000.txt",
      "chars": 359,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2019/08/transfer_learning_methods_small.png",
      "bytes": 312510,
      "reused": true,
      "local_path": "attachments/dba5b30674d7/images/001.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2019/08/transfer_learning_methods_small.png",
      "alt": "The State of Transfer Learning in NLP",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/agenda.png",
      "bytes": 130416,
      "reused": true,
      "local_path": "attachments/dba5b30674d7/images/005.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/agenda.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/transfer_learning_scenario.png",
      "bytes": 116655,
      "reused": true,
      "local_path": "attachments/dba5b30674d7/images/006.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/transfer_learning_scenario.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/ner_results.png",
      "bytes": 299571,
      "reused": true,
      "local_path": "attachments/dba5b30674d7/images/007.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/ner_results.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/transfer_learning_taxonomy.png",
      "bytes": 288373,
      "reused": true,
      "local_path": "attachments/dba5b30674d7/images/008.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/transfer_learning_taxonomy.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/pretraining_adaptation.png",
      "bytes": 120783,
      "reused": true,
      "local_path": "attachments/dba5b30674d7/images/009.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/pretraining_adaptation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/sample_efficiency.png",
      "bytes": 545746,
      "reused": false,
      "local_path": "attachments/dba5b30674d7/images/body-d69d5aa58a38a99734f0.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/sample_efficiency.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/scaling_up_pretraining.png",
      "bytes": 109481,
      "reused": false,
      "local_path": "attachments/dba5b30674d7/images/body-2f61ef89ec419612b1e7.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/scaling_up_pretraining.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/probing_setup.png",
      "bytes": 269828,
      "reused": false,
      "local_path": "attachments/dba5b30674d7/images/body-321f29f6452d8aada4c1.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/probing_setup.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/adapter_layer_small.png",
      "bytes": 192598,
      "reused": false,
      "local_path": "attachments/dba5b30674d7/images/body-42824a47fdd86a2e3800.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/adapter_layer_small.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/feature_extraction.png",
      "bytes": 373766,
      "reused": false,
      "local_path": "attachments/dba5b30674d7/images/body-fe6500c19aa643fcb43b.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/feature_extraction.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/discriminative_fine_tuning-1.png",
      "bytes": 570104,
      "reused": false,
      "local_path": "attachments/dba5b30674d7/images/body-181d085dbb3a2641dbc6.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/discriminative_fine_tuning-1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/sentence_encoders_on_stilts_5k_examples.png",
      "bytes": 188078,
      "reused": false,
      "local_path": "attachments/dba5b30674d7/images/body-43adb63867f8b03acbdf.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/08/sentence_encoders_on_stilts_5k_examples.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.ruder.io/tag/transfer-learning/",
      "label": "transfer learning",
      "type": "a"
    },
    {
      "href": "https://www.ruder.io/author/sebastian/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.infoq.cn/article/zD5QkcIzF9253friWPVd?ref=ruder.io",
      "label": "Chinese",
      "type": "a"
    },
    {
      "href": "https://www.ibidemgroup.com/edu/traduccion-aprendizaje-pnl/?ref=ruder.io",
      "label": "Spanish",
      "type": "a"
    },
    {
      "href": "http://tiny.cc/NAACLTransfer?ref=ruder.io",
      "label": "NAACL 2019 tutorial on Transfer Learning in NLP",
      "type": "a"
    },
    {
      "href": "http://tiny.cc/NAACLTransferColab?ref=ruder.io",
      "label": "Colaboratory notebook",
      "type": "a"
    },
    {
      "href": "http://tiny.cc/NAACLTransferCode?ref=ruder.io",
      "label": "code",
      "type": "a"
    },
    {
      "href": "http://ruder.io/transfer-learning/?ref=ruder.io",
      "label": "this blog post",
      "type": "a"
    },
    {
      "href": "https://thegradient.pub/nlp-imagenet/?ref=ruder.io",
      "label": "become ubiquitous in NLP",
      "type": "a"
    },
    {
      "href": "http://www.jmlr.org/papers/volume6/ando05a/ando05a.pdf?ref=ruder.io",
      "label": "Ando and Zhang, 2005",
      "type": "a"
    },
    {
      "href": "https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35520.pdf?ref=ruder.io",
      "label": "Lin and Wu, 2009",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1705.00108.pdf?ref=ruder.io",
      "label": "Peters et al., 2017",
      "type": "a"
    },
    {
      "href": "https://aclweb.org/anthology/N18-1202?ref=ruder.io",
      "label": "Peters et al., 2018",
      "type": "a"
    },
    {
      "href": "https://alanakbik.github.io/papers/coling2018.pdf?ref=ruder.io",
      "label": "Akbik et al., 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1903.07785?ref=ruder.io",
      "label": "Baevski et al., 2019",
      "type": "a"
    },
    {
      "href": "http://ruder.io/thesis/neural_transfer_learning_for_nlp.pdf?ref=ruder.io",
      "label": "Ruder, 2019",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf?ref=ruder.io",
      "label": "Mikolov et al., 2013",
      "type": "a"
    },
    {
      "href": "https://cs.stanford.edu/~quocle/paragraph_vector.pdf?ref=ruder.io",
      "label": "Le and Mikolov, 2014",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1705.02364?ref=ruder.io",
      "label": "Conneau et al., 2017",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/7209-learned-in-translation-contextualized-word-vectors.pdf?ref=ruder.io",
      "label": "McCann et al., 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1609.08144?ref=ruder.io",
      "label": "Wu et al., 2016",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1809.10040?ref=ruder.io",
      "label": "Zhang et al. 2018",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1439?ref=ruder.io",
      "label": "Wang et al., 2019",
      "type": "a"
    },
    {
      "href": "http://socsci.uci.edu/~rfutrell/papers/hahn2019estimating.pdf?ref=ruder.io",
      "label": "Hahn and Futrell, 2019",
      "type": "a"
    },
    {
      "href": "https://aclweb.org/anthology/P18-1031?ref=ruder.io",
      "label": "Howard and Ruder, 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1907.12412?ref=ruder.io",
      "label": "ERNIE 2.0",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1906.08237?ref=ruder.io",
      "label": "XLNet",
      "type": "a"
    },
    {
      "href": "https://devblogs.nvidia.com/training-bert-with-gpus/?ref=ruder.io",
      "label": "GPT-2 8B",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1907.11692?ref=ruder.io",
      "label": "RoBERTa",
      "type": "a"
    },
    {
      "href": "https://www.jair.org/index.php/jair/article/view/11640?ref=ruder.io",
      "label": "Ruder et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N19-1162?ref=ruder.io",
      "label": "Schuster et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N19-1423?ref=ruder.io",
      "label": "Devlin et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1812.10464?ref=ruder.io",
      "label": "Artetxe and Schwenk, 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N19-1392?ref=ruder.io",
      "label": "Mulcaire et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1901.07291?ref=ruder.io",
      "label": "Lample and Conneau, 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1027?ref=ruder.io",
      "label": "Heinzerling and Strube, 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1493?ref=ruder.io",
      "label": "Pires et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1904.09077?ref=ruder.io",
      "label": "Wu and Dredze, 2019",
      "type": "a"
    },
    {
      "href": "http://ruder.io/publications/?ref=ruder.io",
      "label": "Eisenschlos et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.mitpressjournals.org/doi/pdfplus/10.1162/tacl_a_00019?ref=ruder.io",
      "label": "Williams et al. 2018",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1337?ref=ruder.io",
      "label": "Kuncoro et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.05950?ref=ruder.io",
      "label": "Tenney et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1901.05287?ref=ruder.io",
      "label": "Goldberg, 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N19-1112?ref=ruder.io",
      "label": "Liu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1902.00751?ref=ruder.io",
      "label": "Houlsby et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1902.02671?ref=ruder.io",
      "label": "Stickland and Murray, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1802.05365?ref=ruder.io",
      "label": "Peters et al., 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1705.08142?ref=ruder.io",
      "label": "Ruder et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.sciencedirect.com/science/article/pii/S0079742108605368?ref=ruder.io",
      "label": "McCloskey & Cohen, 1989",
      "type": "a"
    },
    {
      "href": "http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.480.7627&rep=rep1&type=pdf&ref=ruder.io",
      "label": "French, 1999",
      "type": "a"
    },
    {
      "href": "https://www.cs.toronto.edu/~hinton/absps/fastnc.pdf?ref=ruder.io",
      "label": "Hinton et al., 2006",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/3048-greedy-layer-wise-training-of-deep-networks.pdf?ref=ruder.io",
      "label": "Bengio et al., 2007",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D17-1169?ref=ruder.io",
      "label": "Felbo et al., 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1801.06146?ref=ruder.io",
      "label": "Howard and Ruder, 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1902.10547?ref=ruder.io",
      "label": "Chronopoulou et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1908.03265?ref=ruder.io",
      "label": "Liu et al. (2019)",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/K17-1029?ref=ruder.io",
      "label": "Wiese et al., CoNLL 2017",
      "type": "a"
    },
    {
      "href": "https://www.pnas.org/content/114/13/3521?ref=ruder.io",
      "label": "Kirkpatrick et al., PNAS 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1903.05987?ref=ruder.io",
      "label": "Peters et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1811.01088?ref=ruder.io",
      "label": "Phang et al., 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1811.01088v2?ref=ruder.io",
      "label": "Phang et al., 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1901.11373?ref=ruder.io",
      "label": "Yogatama et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1704.07156?ref=ruder.io",
      "label": "Rei et al., 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1901.11504?ref=ruder.io",
      "label": "Liu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://dawn.cs.stanford.edu/2019/03/22/glue/?ref=ruder.io",
      "label": "auxiliary heads that are trained only on particular subsets of the data",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1809.08370?ref=ruder.io",
      "label": "Clark et al., 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1904.12848?ref=ruder.io",
      "label": "Xie et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1907.04829?ref=ruder.io",
      "label": "Clark et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1355?ref=ruder.io",
      "label": "Strubell et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.tensorflow.org/hub?ref=ruder.io",
      "label": "TensorFlow Hub",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/hub?ref=ruder.io",
      "label": "PyTorch Hub",
      "type": "a"
    },
    {
      "href": "https://allennlp.org/?ref=ruder.io",
      "label": "AllenNLP",
      "type": "a"
    },
    {
      "href": "https://github.com/fastai/fastai?ref=ruder.io",
      "label": "fast.ai",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/pytorch-transformers?ref=ruder.io",
      "label": "pytorch-transformers",
      "type": "a"
    },
    {
      "href": "https://docs.google.com/presentation/d/1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc/edit?ref=ruder.io",
      "label": "code snippets",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1903.08855?ref=ruder.io",
      "label": "Liu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.07830?ref=ruder.io",
      "label": "Zellers et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1907.10529?ref=ruder.io",
      "label": "Joshi et al., 2019",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/1905.07129?ref=ruder.io",
      "label": "Zhang et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1598?ref=ruder.io",
      "label": "Logan IV et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1904.01766?ref=ruder.io",
      "label": "Sun et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1908.02265?ref=ruder.io",
      "label": "Lu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N19-5004?ref=ruder.io",
      "label": "the tutorial",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:54Z",
  "source_url": "https://www.ruder.io/state-of-transfer-learning-in-nlp/",
  "papers": [
    {
      "status": "ok",
      "url": "http://www.jmlr.org/papers/volume6/ando05a/ando05a.pdf?ref=ruder.io",
      "bytes": 401640,
      "reused": false,
      "local_path": "papers/repair/a7821668cb91e17de00b.pdf"
    },
    {
      "status": "ok",
      "url": "https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35520.pdf?ref=ruder.io",
      "bytes": 225980,
      "reused": false,
      "local_path": "papers/repair/0eb5f09b5164aa73142d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1705.00108.pdf?ref=ruder.io",
      "bytes": 401085,
      "reused": false,
      "local_path": "papers/repair/649df57aba9688fee144.pdf"
    },
    {
      "status": "ok",
      "url": "https://alanakbik.github.io/papers/coling2018.pdf?ref=ruder.io",
      "bytes": 1831991,
      "reused": false,
      "local_path": "papers/repair/52c78494ca5babc0da6b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1903.07785",
      "bytes": 277752,
      "reused": false,
      "local_path": "papers/repair/a4200e001b92f5e311c0.pdf"
    },
    {
      "status": "http_522",
      "url": "http://ruder.io/thesis/neural_transfer_learning_for_nlp.pdf?ref=ruder.io",
      "retried_at": "2026-10-20T17:49:52Z"
    },
    {
      "status": "ok",
      "url": "https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf?ref=ruder.io",
      "bytes": 111997,
      "reused": false,
      "local_path": "papers/repair/a5a0181b47b602bf4dc2.pdf"
    },
    {
      "status": "ok",
      "url": "https://cs.stanford.edu/~quocle/paragraph_vector.pdf?ref=ruder.io",
      "bytes": 292960,
      "reused": false,
      "local_path": "papers/repair/6c1f5905c5140cbf5342.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1705.02364",
      "bytes": 473315,
      "reused": false,
      "local_path": "papers/repair/3418178c044796223ad7.pdf"
    },
    {
      "status": "ok",
      "url": "http://papers.nips.cc/paper/7209-learned-in-translation-contextualized-word-vectors.pdf?ref=ruder.io",
      "bytes": 412761,
      "reused": false,
      "local_path": "papers/repair/248cb3f1a47e3cfad8d4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1609.08144",
      "bytes": 1688289,
      "reused": true,
      "local_path": "papers/repair/f8e7010861b5c08375c8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1809.10040",
      "bytes": 821317,
      "reused": false,
      "local_path": "papers/repair/a968384934c8d1d9832a.pdf"
    },
    {
      "status": "ok",
      "url": "http://socsci.uci.edu/~rfutrell/papers/hahn2019estimating.pdf?ref=ruder.io",
      "bytes": 1314102,
      "reused": false,
      "local_path": "papers/repair/2735de8e1870ef50a5b1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1907.12412",
      "bytes": 433492,
      "reused": false,
      "local_path": "papers/repair/ab12479c58a1c19d2b4e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1906.08237",
      "bytes": 761790,
      "reused": true,
      "local_path": "papers/repair/e1dc08bfcdc2d830fa23.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1907.11692",
      "bytes": 209675,
      "reused": true,
      "local_path": "papers/repair/8acd467c11db68744d3f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1812.10464",
      "bytes": 380205,
      "reused": true,
      "local_path": "papers/repair/6b93f182a2711042593b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1901.07291",
      "bytes": 342326,
      "reused": true,
      "local_path": "papers/repair/5f13ba6b638f9c631c3d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.09077",
      "bytes": 469580,
      "reused": false,
      "local_path": "papers/repair/49138dc5e38c002441a1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.05950",
      "bytes": 1511448,
      "reused": true,
      "local_path": "papers/repair/4643628d574171655bf9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1901.05287",
      "bytes": 87208,
      "reused": false,
      "local_path": "papers/repair/d966319969744769b9b5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1902.00751",
      "bytes": 872475,
      "reused": true,
      "local_path": "papers/repair/d18a13082d88db0c358a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1902.02671",
      "bytes": 406998,
      "reused": false,
      "local_path": "papers/repair/892b6999712e547b1683.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1802.05365",
      "bytes": 425880,
      "reused": true,
      "local_path": "papers/repair/c4dc3d740cad43df1dd2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1705.08142",
      "bytes": 395051,
      "reused": false,
      "local_path": "papers/repair/f39795e74a3c15abab20.pdf"
    },
    {
      "status": "ok",
      "url": "https://www.cs.toronto.edu/~hinton/absps/fastnc.pdf?ref=ruder.io",
      "bytes": 633269,
      "reused": false,
      "local_path": "papers/repair/3468e1f1d7bb48b45037.pdf"
    },
    {
      "status": "ok",
      "url": "https://papers.nips.cc/paper/3048-greedy-layer-wise-training-of-deep-networks.pdf?ref=ruder.io",
      "bytes": 187377,
      "reused": false,
      "local_path": "papers/repair/c2b7990e36e1f2494b21.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1801.06146",
      "bytes": 979417,
      "reused": true,
      "local_path": "papers/repair/f99f602eebf2965d0ed0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1902.10547",
      "bytes": 1223329,
      "reused": false,
      "local_path": "papers/repair/8573f8c5498e6b0d226a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1908.03265",
      "bytes": 2872984,
      "reused": false,
      "local_path": "papers/repair/f29259c672c8c60a4863.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1903.05987",
      "bytes": 548329,
      "reused": false,
      "local_path": "papers/repair/7bfce21923a0bbd89dcc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1811.01088",
      "bytes": 323286,
      "reused": true,
      "local_path": "papers/repair/900b122b4c467abf7740.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1811.01088v2",
      "bytes": 323286,
      "reused": false,
      "local_path": "papers/repair/c5cbb012dbde29b6bcdb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1901.11373",
      "bytes": 1034653,
      "reused": true,
      "local_path": "papers/repair/a65e297158fc2c17bcf7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1704.07156",
      "bytes": 458047,
      "reused": false,
      "local_path": "papers/repair/7e1563bf12b5a58e4649.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1901.11504",
      "bytes": 483541,
      "reused": false,
      "local_path": "papers/repair/79f73ba94566b8c3abda.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1809.08370",
      "bytes": 811545,
      "reused": false,
      "local_path": "papers/repair/459221e6be389ff8d946.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.12848",
      "bytes": 1625724,
      "reused": true,
      "local_path": "papers/repair/17af125202b45606d9ad.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1907.04829",
      "bytes": 272469,
      "reused": false,
      "local_path": "papers/repair/a74b4f5893bfb0465c00.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1903.08855",
      "bytes": 791761,
      "reused": false,
      "local_path": "papers/repair/4d4b013fbf18a9475c76.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.07830",
      "bytes": 1192712,
      "reused": false,
      "local_path": "papers/repair/e073c507daa9cf32540f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1907.10529",
      "bytes": 1532033,
      "reused": false,
      "local_path": "papers/repair/62d48c1f8a46f02fa94c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.07129",
      "bytes": 1787781,
      "reused": false,
      "local_path": "papers/repair/f871456878c517a6a7a2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.01766",
      "bytes": 6264960,
      "reused": true,
      "local_path": "papers/repair/d1fc0ae2f1fd2c4c6300.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1908.02265",
      "bytes": 5606833,
      "reused": false,
      "local_path": "papers/repair/ecec80af8d330dea1383.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 23064,
  "expected_body_images": 13,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "http_522",
  "raw_html": "attachments/dba5b30674d7/pages/source.html",
  "local_html": "attachments/dba5b30674d7/pages/article.html",
  "body_base_path": "attachments/dba5b30674d7/pages/",
  "body_markdown_path": "attachments/dba5b30674d7/pages/article.md"
}