{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Sebastian Ruder",
  "title_listed": "10 ML & NLP Research Highlights of 2019",
  "keywords": "研究回顾,ML,NLP",
  "domain": "机器学习",
  "content_type": "行业分析",
  "is_paid_flag": "否",
  "url": "https://www.ruder.io/research-highlights-2019/",
  "publish_date": "2020-01",
  "corpus_stage": "预训练/SFT",
  "notes": "2019年ML与NLP研究十大亮点",
  "id": "202398738caa",
  "collected_at": "2026-10-02T14:16:20",
  "title": "10 ML & NLP Research Highlights of 2019",
  "body_markdown": "natural language processing\n10 ML & NLP Research Highlights of 2019\nThis post gathers ten ML and NLP research directions that I found exciting and impactful in 2019.\nSebastian Ruder\n06 Jan 2020\n•\n12 min read\nShare\nThis post gathers ten ML and NLP research directions that I found exciting and impactful in 2019.\nFor each highlight, I summarise the main advances that took place this year, briefly state why I think it is important, and provide a short outlook to the future.\nThe full list of highlights is here:\nUniversal unsupervised pretraining\nLottery tickets\nThe Neural Tangent Kernel\nUnsupervised multilingual learning\nMore robust benchmarks\nML and NLP for science\nFixing decoding errors in NLG\nAugmenting pretrained models\nEfficient and long-range Transformers\nMore reliable analysis methods\n1) Universal unsupervised pretraining\nWhat happened?\nUnsupervised pretraining was prevalent in NLP this year, mainly driven by BERT (\nDevlin et al., 2019\n) and other variants.\nA whole range of BERT variants\nhave been applied to multimodal settings, mostly involving images and videos together with text (for an example see the figure below). Unsupervised pretraining has also made inroads into domains where supervision had previously reigned supreme. In biology, Transformer language models have been pretrained on protein sequences (\nRives et al., 2019\n). In computer vision, approaches leveraged self-supervision including CPC (\nHénaff et al., 2019\n), MoCo (\nHe et al., 2019\n), and PIRL (\nMisra & van der Maaten, 2019\n) as well as strong generators such as BigBiGAN (\nDonahue & Simonyan, 2019\n) to improve sample efficiency on ImageNet and image generation. In speech, representations learned with a multi-layer CNN (\nSchneider et al., 2019\n) or bidirectional CPC (\nKawakami et al., 2019\n) outperform state-of-the-art models with much less training data.\nWhy is it important?\nUnsupervised pretraining enables training models with much fewer labelled examples. This opens up new applications in many different domains where data requirements were previously prohibitive.\nWhat's next?\nUnsupervised pretraining is here to stay. While the biggest advances have been achieved so far in individual domains, it will be interesting to see a focus towards tighter integration of multiple modalities.\nVideoBERT (\nSun et al., 2019\n), a recent multimodal variant of BERT that generates video \"tokens\" given a recipe (above) and predicts future tokens at different time scales given a video token (below).\n2) Lottery tickets\nWhat happened?\nFrankle and Carbin\n(2019) identified\nwinning tickets\n, subnetworks in dense, randomly-initialised, feed-forward networks that are so well initialised that training them in isolation achieves similar accuracy to training the full network, as can be seen below. While the initial pruning procedure only worked on small vision tasks, later work (\nFrankle et al., 2019\n) applied the pruning early in training instead of at initialisation, which makes it possible to find small subnetworks of deeper models.\nYu et al. (2019)\nfind winning ticket initialisations also for LSTMs and Transformers in NLP and RL models. While winning tickets are still expensive to find, it is promising that they seem to be transferable across datasets and optimisers (\nMorcos et al., 2019\n) and domains (\nDesai et al., 2019\n).\nWhy is it important?\nState-of-the-art neural networks are getting larger and more expensive to train and to use for prediction. Being able to consistently identify small subnetworks that achieve comparable performance enables training and inference with much fewer resources. This can speed up model iteration and opens up new applications in on-device and edge computing.\nWhat's next?\nIdentifying winning tickets is currently still too expensive to provide real benefits in low-resource settings. More robust one-shot pruning methods that are less susceptible to noise in the pruning process should mitigate this. Investigating what makes winning tickets special should also help us gain a better understanding of the initialisation and learning dynamics of neural networks.\nTest accuracy of winning tickets (solid lines) vs. randomly sampled subnetworks (dashed lines) at different pruning ratios (\nFrankle & Carbin, 2019\n).\n3) The Neural Tangent Kernel\nWhat happened?\nSomewhat counter-intuitively, very wide (more concretely,\ninfinitely\nwide) neural networks are easier to study theoretically than narrow ones. It has been shown that in the infinite-width limit, neural networks can be approximated as linear models with a kernel, the Neural Tangent Kernel (NTK;\nJacot et al., 2018\n). Refer to\nthis post\nfor an intuitive explanation of NTK including an illustration of its training dynamics (see the figure below). In practice, such models, have underperformed their finite-depth counterparts (\nNovak et al., 2019\n;\nAllen-Zhu et al., 2019\n;\nBietti & Mairal, 2019\n), which limits applying the findings to standard methods. Recent work (\nLi et al., 2019\n;\nArora et al., 2019\n), however, has significantly reduced the performance gap to standard methods (see\nChip Huyen's post\nfor other related NeurIPS 2019 papers).\nWhy is it important?\nThe NTK is perhaps the most powerful tool at our disposal to analyse the theoretical behaviour of neural networks. While it has its limitations, i.e. practical neural networks still perform better than their NTK counterparts, and insights so far have not translated into empirical gains, it may help us open the black box of deep learning.\nWhat's next?\nThe gap to standard methods seems to be mainly due to benefits of the finite width of such methods, which future work may seek to characterise. This will hopefully also help translating insights from the infinite-width limit to practical settings. Ultimately, the NTK may help us shed light on the training dynamics and generalisation behaviour of neural networks.\nLearning dynamics of linear models with an NTK with different α factors. NTKs are visualised as ellipses (credit:\nRajat's Blog\n).\n4) Unsupervised multilingual learning\nWhat happened?\nCross-lingual representations had mostly focused on the word level for many years (\nsee this survey\n). Building on advances in unsupervised pretraining, this year saw the development of deep cross-lingual models such as\nmultilingual BERT\n, XLM (\nConneau & Lample, 2019\n), and XLM-R (\nConneau et al., 2019\n). Even though these models do not use any explicit cross-lingual signal, they generalise surprisingly well across languages—even without a shared vocabulary or joint training (\nArtetxe et al., 2019\n;\nKarthikeyan et al., 2019\n;\nWu et al., 2019\n). For an overview, have a look at\nthis post\n. Such deep models also brought improvements in unsupervised MT (\nSong et al., 2019\n;\nConneau & Lample, 2019\n), which hit its stride last year (see\nhighlights of 2018\n) and saw improvements from a more principled combination of statistical and neural approaches (\nArtetxe et al., 2019\n). Another exciting development is the bootstrapping of deep multilingual models from readily available pretrained representations in English (\nArtetxe et al., 2019\n;\nTran, 2020\n), which can be seen below.\nWhy is it important?\nReady-to-use cross-lingual representations enable training of models with fewer examples for languages other than English. Furthermore, if labelled data in English is available, these methods enable essentially free zero-shot transfer. They may finally help us gain a better understanding of the relationships between different languages.\nWhat's next?\nIt is still unclear why these methods work so well without any cross-lingual supervision. Gaining a better understanding of how these methods work will likely enable us to design more powerful methods and may also reveal insights about the structure of different languages. In addition, we should not only focus on zero-shot transfer but also consider learning from few labelled examples in the target language (see\nthis post\n).\nThe four steps of the monolingual transfer approach of\nArtetxe et al. (2019)\n5) More robust benchmarks\nThere is something rotten in the state of the art.\n—\nNie et al. (2019)\nparaphrasing\nShakespeare\nWhat happened?\nRecent NLP datasets such as HellaSWAG (\nZellers et al., 2019\n) are created to be difficult for state-of-the-art models to solve. Examples are filtered by humans to explicitly retain only those where state-of-the-art models fail (see below for an example). This process of human-in-the-loop adversarial curation can be repeated multiple times such as in the recent Adversarial NLI (\nNie et al., 2019\n) benchmark to enable the creation of datasets that are much more challenging for current methods.\nWhy is it important?\nMany researchers have observed that current NLP models do not learn what they are supposed to but instead adopt shallow heuristics and exploit superficial cues in the data (described as\nthe Clever Hans moment\n). As datasets become more robust, we would hope that models will be forced to eventually learn the true underlying relations in the data.\nWhat's next?\nAs models become better, most datasets will need to be continuously improved or will quickly become outdated. Dedicated infrastructure and tools will be necessary to facilitate this process. In addition, appropriate baselines should be run including simple methods and models using different variants of the data (such as with incomplete input) so that initial versions of datasets are as robust as possible.\nA multiple-choice sentence completion example from HellaSWAG that is difficult to answer for state-of-the-art models. Most hard examples lie in a \"Goldilocks zone\" of complexity, consisting roughly of three sentences of context and two generated sentences (\nZellers et al., 2019\n).\n6) ML and NLP for science\nWhat happened?\nThere have been some major advances of ML being applied to fundamental science problems. My highlights were the application of deep neural networks to\nprotein folding\nand to the Many-Electron Schrödinger Equation (\nPfau et al., 2019\n). On the NLP side, it is exciting to see what impact even standard methods can have when combined with domain expertise. One study used word embeddings to analyse latent knowledge in the materials science literature (\nTshitoyan et al., 2019\n), which can be used to predict which materials will have certain properties (see the figure below). In biology, a lot of data such as genes and proteins is sequential in nature. It is thus a natural fit for NLP methods such as LSTMs and Transformers, which have been applied to protein classification (\nStrodthoff et al., 2019\n;\nRives et al., 2019\n).\nWhy is it important?\nScience is arguably one of the most impactful application domains for ML. Solutions can have a large impact on many other domains and can help solve real-world problems.\nWhat's next?\nFrom modelling energy in physics problems (\nGreydanus et al., 2019\n) to solving differential equations (\nLample & Charton, 2020\n), ML methods are constantly being applied to new applications in science. It will be interesting to see what the most impactful of these will be in 2020.\nUsing word embeddings trained on abstracts from different time periods to predict which materials will be studied as ferroelectric (a), photovoltaics (b), and topological insulator (c) in future abstracts. Top 50 predictions are much more likely to be studied compared to all candidate materials (\nTshitoyan et al., 2019\n).\n7) Fixing decoding errors in NLG\nWhat happened?\nDespite ever more powerful models, natural language generation (NLG) models still frequently produce repetitions or gibberish as can be seen below. This was shown to be mainly a result of the maximum likelihood training. I was excited to see improvements that aim to ameliorate this and are orthogonal to advances in modelling. Such improvements came in the form of new sampling methods, such as nucleus sampling (\nHoltzman et al., 2019\n) and new loss functions (\nWelleck et al., 2019\n). Another surprising finding was that better search does not lead to better generations: Current models rely to some extent on an imperfect search and beam search errors. In contrast, an exact search most often returns an empty translation in the case of machine translation (\nStahlberg & Byrne, 2019\n). This shows that advances in search and modelling must often go hand in hand.\nWhy is it important?\nNatural language generation is one of the most general tasks in NLP. In NLP and ML research, most papers focus on improving the model, while other parts of the pipeline are typically neglected. For NLG, it is important to remind ourselves that our models still have flaws and that it may be possible to improve the output by fixing the search or the training process.\nWhat's next?\nDespite more powerful models and successful applications of transfer learning to NLG (\nSong et al., 2019\n;\nWolf et al., 2019\n), model predictions still contain many artefacts. Identifying and understanding the causes of such artefacts is an important research direction.\nRepetitions (blue) and gibberish (red) produced by GPT-2 using beam search and pure (greedy) sampling (\nHoltzman et al., 2019\n).\n8) Augmenting pretrained models\nWhat happened?\nI was excited to see approaches that equip pretrained models with new capabilities. Some methods augment a pretrained model with a knowledge base in order to improve modelling of entity names (\nLiu et al., 2019\n) and the recall of facts (\nLogan et al., 2019\n). Others enable it to perform simple arithmetic reasoning (\nAndor et al., 2019\n) by giving it access to a number of predefined executable programs. As most models have a weak inductive bias and learn most of their knowledge from data, another way to extend a pretrained model is by augmenting the training data itself, e.g. to capture common sense (\nBosselut et al., 2019\n) as can be seen below.\nWhy is it important?\nModels are becoming more powerful but\nthere are many things\nthat a model cannot learn from text alone. Particularly when dealing with more complex tasks, the available data may be too limited to learn explicit reasoning using facts or common sense and a stronger inductive bias may often be necessary.\nWhat's next?\nAs models are being applied to more challenging problems, it will increasingly become necessary for modifications to be compositional. In the future, we might combine powerful pretrained models with learnable compositional programs (\nPierrot et al., 2019\n).\nA standard Transformer with multi-head attention. The model is trained to predict the object of a knowledge base triple given its subject and relation (\nBosselut et al., 2019\n).\n9) Efficient and long-range Transformers\nWhat happened?\nThis year saw several improvements to the Transformer (\nVaswani et al., 2017\n) architecture. The Transformer-XL (\nDai et al., 2019\n) and the Compressive Transformer (\nRae et al., 2020\n), which can be seen below enable it to better capture long-range dependencies. Many approaches sought to make the Transformer more efficient mostly using different—often sparse—attention mechanisms, such as adaptively sparse attention (\nCorreia et al., 2019\n), adaptive attention spans (\nSukhbaatar et al., 2019\n), product-key attention (\nLample et al., 2019)\n, and locality-sensitive hashing (\nKitaev et al., 2020\n). On the Transformer-based pretraining front, there have been more efficient variants such as ALBERT (\nLan et al., 2020\n), which employs parameter sharing and ELECTRA (\nClark et al., 2020\n), which uses a more efficient pretraining task. There were also more efficient pretrained models that did not utilise a Transformer, such as the unigram document model VAMPIRE (\nGururangan et al., 2019\n) and the QRNN-based MultiFiT (\nEisenschlos et al., 2019\n). Another trend was the distillation of large BERT models into smaller ones (\nTang et al., 2019\n;\nTsai et al., 2019\n;\nSanh et al., 2019\n).\nWhy is it important?\nThe Transformer architecture has been influential since its inception. It is a part of most state-of-the-art models in NLP and has been successfully applied to many other domains (see Sections\n1\nand\n6\n). Any improvement to the Transformer architecture may thus have strong ripple effects.\nWhat's next?\nIt will take some time for these improvements to trickle down to the practitioner but given the prevalence and ease of use of pretrained models, more efficient alternatives will likely be adopted quickly. Overall, we will hopefully see a continuing focus on model architectures that emphasise efficiency, with sparsity being one of the key trends.\nThe Compressive Transformer compresses (a fine-grained memory of) past activations into a coarser compressed memory (\nRae et al., 2020\n).\n10) More reliable analysis methods\nWhat happened?\nA key trend for me this year was the increasing number of papers analysing models. In fact, several of my favourite papers this year were such analysis papers. An early highlight was the excellent survey of analysis methods by\nBelinkov & Glass (2019)\n. This year was also the first one (in my memory) where many papers were dedicated to analysing a single model, BERT (such papers are known as\nBERTology\n). In this context, probes (see the figure below), which aim to understand whether a model captures morphology, syntax, etc. by predicting certain properties have become a common tool. I particularly appreciated papers that make probes more reliable (\nLiu et al., 2019\n;\nHewitt & Liang, 2019\n). Reliability is also a theme in the ongoing conversation on whether attention provides meaningful explanations (\nJain & Wallace, 2019\n;\nWiegreffe & Pinter, 2019\n;\nWallace, 2019\n). The continuing interest in analysis methods is perhaps best exemplified by the new ACL 2020 track on\nInterpretability and Analysis of Models in NLP\n.\nWhy is it important?\nState-of-the-art methods are used as black boxes. In order to develop better models and to use them in the real world, we need to understand why models make certain decisions. However, our current methods to explain models' predictions are still limited.\nWhat's next?\nWe need more work on explaining predictions that goes beyond visualisation, which is often unreliable. An important trend in this direction are human-written explanations that are being provided by more datasets (\nCamburu et al., 2018\n;\nRajani et al., 2019\n;\nNie et al., 2019\n).\nThe probing setup used to study linguistic knowledge in representations (\nLiu et al., 2019\n).",
  "metadata": {
    "title": "10 ML & NLP Research Highlights of 2019",
    "description": "This post gathers ten ML and NLP research directions that I found exciting and impactful in 2019.",
    "keywords": null,
    "author": null,
    "og:site_name": "ruder.io",
    "og:type": "article",
    "og:title": "10 ML & NLP Research Highlights of 2019",
    "og:description": "This post gathers ten ML and NLP research directions that I found exciting and impactful in 2019.",
    "og:url": "https://www.ruder.io/research-highlights-2019/",
    "og:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2020/01/videobert-1.png",
    "og:image:width": "1200",
    "og:image:height": "437",
    "twitter:card": "summary_large_image",
    "twitter:title": "10 ML & NLP Research Highlights of 2019",
    "twitter:description": "This post gathers ten ML and NLP research directions that I found exciting and impactful in 2019.",
    "twitter:url": "https://www.ruder.io/research-highlights-2019/",
    "twitter:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2020/01/videobert-1.png",
    "twitter:label1": "Written by",
    "twitter:data1": "Sebastian Ruder",
    "twitter:label2": "Filed under",
    "twitter:data2": "natural language processing, cross-lingual, transfer learning",
    "twitter:site": "@seb_ruder",
    "article:published_time": "2020-01-06T08:00:00.000Z",
    "article:modified_time": "2020-01-13T23:28:49.000Z",
    "article:tag": "transfer learning"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2020/01/videobert-1.png",
      "bytes": 1330321,
      "reused": true,
      "local_path": "attachments/202398738caa/images/001.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2020/01/videobert-1.png",
      "alt": "10 ML & NLP Research Highlights of 2019",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/videobert.png",
      "bytes": 1838944,
      "reused": true,
      "local_path": "attachments/202398738caa/images/005.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/videobert.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/winning_tickets.png",
      "bytes": 93000,
      "reused": true,
      "local_path": "attachments/202398738caa/images/006.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/winning_tickets.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/ntk_dynamics.gif",
      "bytes": 3367763,
      "reused": false,
      "local_path": "attachments/202398738caa/images/body-7a9a0f53133ae0226283.gif",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/ntk_dynamics.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/monolingual_transfer.png",
      "bytes": 410433,
      "reused": false,
      "local_path": "attachments/202398738caa/images/body-87d8fc9a55e03b1618c5.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/monolingual_transfer.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/hellaswag.png",
      "bytes": 334071,
      "reused": false,
      "local_path": "attachments/202398738caa/images/body-c989f1dad3e494751449.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/hellaswag.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/historical_validations_functional_material_predictions.png",
      "bytes": 404624,
      "reused": false,
      "local_path": "attachments/202398738caa/images/body-cc6d4f5b3cb8a2427075.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/historical_validations_functional_material_predictions.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/text_degeneration.png",
      "bytes": 379716,
      "reused": false,
      "local_path": "attachments/202398738caa/images/body-5f4359da1809168d2783.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/text_degeneration.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/comet.png",
      "bytes": 140236,
      "reused": false,
      "local_path": "attachments/202398738caa/images/body-917b5ef64456777823a8.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/comet.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/compressive_transformer.png",
      "bytes": 223815,
      "reused": false,
      "local_path": "attachments/202398738caa/images/body-d028116e7873cdabedf0.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/compressive_transformer.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/probing_landscape.png",
      "bytes": 152560,
      "reused": false,
      "local_path": "attachments/202398738caa/images/body-eca95054348700523dfd.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2020/01/probing_landscape.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.ruder.io/tag/natural-language-processing/",
      "label": "natural language processing",
      "type": "a"
    },
    {
      "href": "https://www.ruder.io/author/sebastian/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N19-1423/?ref=ruder.io",
      "label": "Devlin et al., 2019",
      "type": "a"
    },
    {
      "href": "https://twitter.com/xwang_lk/status/1166187855456002049?ref=ruder.io",
      "label": "A whole range of BERT variants",
      "type": "a"
    },
    {
      "href": "https://www.biorxiv.org/content/10.1101/622803v1.full?ref=ruder.io",
      "label": "Rives et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.09272?ref=ruder.io",
      "label": "Hénaff et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1911.05722?ref=ruder.io",
      "label": "He et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.ruder.io/research-highlights-2019/arxiv.org/abs/1912.01991",
      "label": "Misra & van der Maaten, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1907.02544?ref=ruder.io",
      "label": "Donahue & Simonyan, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1904.05862?ref=ruder.io",
      "label": "Schneider et al., 2019",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=HJe-blSYvH&ref=ruder.io",
      "label": "Kawakami et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1904.01766?ref=ruder.io",
      "label": "Sun et al., 2019",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=rJl-b3RcF7&ref=ruder.io",
      "label": "Frankle and Carbin",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1903.01611?ref=ruder.io",
      "label": "Frankle et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1906.02768?ref=ruder.io",
      "label": "Yu et al. (2019)",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/8739-one-ticket-to-win-them-all-generalizing-lottery-ticket-initializations-across-datasets-and-optimizers.pdf?ref=ruder.io",
      "label": "Morcos et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D19-6117/?ref=ruder.io",
      "label": "Desai et al., 2019",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/8076-neural-tangent-kernel-convergence-and-generalization-in-neural-networks.pdf?ref=ruder.io",
      "label": "Jacot et al., 2018",
      "type": "a"
    },
    {
      "href": "https://rajatvd.github.io/NTK/?ref=ruder.io",
      "label": "this post",
      "type": "a"
    },
    {
      "href": "https://openreview.net/pdf?id=B1g30j0qF7&ref=ruder.io",
      "label": "Novak et al., 2019",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/8847-learning-and-generalization-in-overparameterized-neural-networks-going-beyond-two-layers?ref=ruder.io",
      "label": "Allen-Zhu et al., 2019",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/9449-on-the-inductive-bias-of-neural-tangent-kernels?ref=ruder.io",
      "label": "Bietti & Mairal, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1911.00809?ref=ruder.io",
      "label": "Li et al., 2019",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/9025-on-exact-computation-with-an-infinitely-wide-neural-net.pdf?ref=ruder.io",
      "label": "Arora et al., 2019",
      "type": "a"
    },
    {
      "href": "https://huyenchip.com/2019/12/18/key-trends-neurips-2019.html?ref=ruder.io",
      "label": "Chip Huyen's post",
      "type": "a"
    },
    {
      "href": "https://www.jair.org/index.php/jair/article/view/11640?ref=ruder.io",
      "label": "see this survey",
      "type": "a"
    },
    {
      "href": "https://github.com/google-research/bert/blob/master/multilingual.md?ref=ruder.io",
      "label": "multilingual BERT",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/8928-cross-lingual-language-model-pretraining.pdf?ref=ruder.io",
      "label": "Conneau & Lample, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1911.02116?ref=ruder.io",
      "label": "Conneau et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1910.11856?ref=ruder.io",
      "label": "Artetxe et al., 2019",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=HJeT3yrtDr&ref=ruder.io",
      "label": "Karthikeyan et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1911.01464?ref=ruder.io",
      "label": "Wu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://ruder.io/unsupervised-cross-lingual-learning/?ref=ruder.io",
      "label": "this post",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.02450?ref=ruder.io",
      "label": "Song et al., 2019",
      "type": "a"
    },
    {
      "href": "https://ruder.io/10-exciting-ideas-of-2018-in-nlp/?ref=ruder.io",
      "label": "highlights of 2018",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1019/?ref=ruder.io",
      "label": "Artetxe et al., 2019",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=Bkle6T4YvB&ref=ruder.io",
      "label": "Tran, 2020",
      "type": "a"
    },
    {
      "href": "http://nlp.fast.ai/classification/2019/09/10/multifit.html?ref=ruder.io",
      "label": "this post",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1910.14599?ref=ruder.io",
      "label": "Nie et al. (2019)",
      "type": "a"
    },
    {
      "href": "http://www.shakespeare-online.com/quickquotes/quickquotehamletdenmark.html?ref=ruder.io",
      "label": "Shakespeare",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1472/?ref=ruder.io",
      "label": "Zellers et al., 2019",
      "type": "a"
    },
    {
      "href": "https://thegradient.pub/nlps-clever-hans-moment-has-arrived/?ref=ruder.io",
      "label": "the Clever Hans moment",
      "type": "a"
    },
    {
      "href": "https://www.nature.com/articles/d41586-019-01357-6?ref=ruder.io",
      "label": "protein folding",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1909.02487?ref=ruder.io",
      "label": "Pfau et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.nature.com/articles/s41586-019-1335-8?ref=ruder.io",
      "label": "Tshitoyan et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.biorxiv.org/content/10.1101/704874v2?ref=ruder.io",
      "label": "Strodthoff et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.biorxiv.org/content/10.1101/622803v2?ref=ruder.io",
      "label": "Rives et al., 2019",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/9672-hamiltonian-neural-networks.pdf?ref=ruder.io",
      "label": "Greydanus et al., 2019",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=S1eZYeHFDS&ref=ruder.io",
      "label": "Lample & Charton, 2020",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=rygGQyrFvH&ref=ruder.io",
      "label": "Holtzman et al., 2019",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=SJeYe0NtvH&ref=ruder.io",
      "label": "Welleck et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D19-1331.pdf?ref=ruder.io",
      "label": "Stahlberg & Byrne, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1901.08149?ref=ruder.io",
      "label": "Wolf et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N19-1117/?ref=ruder.io",
      "label": "Liu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1598/?ref=ruder.io",
      "label": "Logan et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D19-1609/?ref=ruder.io",
      "label": "Andor et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1470/?ref=ruder.io",
      "label": "Bosselut et al., 2019",
      "type": "a"
    },
    {
      "href": "https://docs.google.com/presentation/d/1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc/edit?ref=ruder.io",
      "label": "there are many things",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/9608-learning-compositional-neural-programs-with-recursive-tree-search-and-planning?ref=ruder.io",
      "label": "Pierrot et al., 2019",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/7181-attention-is-all-you-need?ref=ruder.io",
      "label": "Vaswani et al., 2017",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1285/?ref=ruder.io",
      "label": "Dai et al., 2019",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=SylKikSYDH&ref=ruder.io",
      "label": "Rae et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D19-1223/?ref=ruder.io",
      "label": "Correia et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1032/?ref=ruder.io",
      "label": "Sukhbaatar et al., 2019",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/9061-large-memory-layers-with-product-keys?ref=ruder.io",
      "label": "Lample et al., 2019)",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=rkgNKkHtvB&ref=ruder.io",
      "label": "Kitaev et al., 2020",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=H1eA7AEtvS&ref=ruder.io",
      "label": "Lan et al., 2020",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=r1xMH1BtvB&ref=ruder.io",
      "label": "Clark et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1590/?ref=ruder.io",
      "label": "Gururangan et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D19-1572/?ref=ruder.io",
      "label": "Eisenschlos et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1903.12136?ref=ruder.io",
      "label": "Tang et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D19-1374/?ref=ruder.io",
      "label": "Tsai et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1910.01108?ref=ruder.io",
      "label": "Sanh et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.mitpressjournals.org/doi/full/10.1162/tacl_a_00254?ref=ruder.io",
      "label": "Belinkov & Glass (2019)",
      "type": "a"
    },
    {
      "href": "http://newsletter.ruder.io/issues/bert-gpt-2-xlnet-naacl-icml-arxiv-eurnlp-180092?ref=ruder.io",
      "label": "BERTology",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N19-1225/?ref=ruder.io",
      "label": "Liu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D19-1275/?ref=ruder.io",
      "label": "Hewitt & Liang, 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/N19-1357/?ref=ruder.io",
      "label": "Jain & Wallace, 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/D19-1002/?ref=ruder.io",
      "label": "Wiegreffe & Pinter, 2019",
      "type": "a"
    },
    {
      "href": "https://medium.com/@byron.wallace/thoughts-on-attention-is-not-not-explanation-b7799c4c3b24?ref=ruder.io",
      "label": "Wallace, 2019",
      "type": "a"
    },
    {
      "href": "https://acl2020.org/calls/papers/?ref=ruder.io",
      "label": "Interpretability and Analysis of Models in NLP",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/8163-e-snli-natural-language-inference-with-natural-language-explanations?ref=ruder.io",
      "label": "Camburu et al., 2018",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P19-1487/?ref=ruder.io",
      "label": "Rajani et al., 2019",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:03Z",
  "source_url": "https://www.ruder.io/research-highlights-2019/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.09272",
      "bytes": 2411659,
      "reused": true,
      "local_path": "papers/repair/b9006248b176b4b194f1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1911.05722",
      "bytes": 593470,
      "reused": true,
      "local_path": "papers/repair/5c3c001fd908f6dae2c6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1907.02544",
      "bytes": 10686100,
      "reused": false,
      "local_path": "papers/repair/e821b43935904065f496.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.05862",
      "bytes": 300151,
      "reused": false,
      "local_path": "papers/repair/eedf4606147ad7da22e4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.01766",
      "bytes": 6264960,
      "reused": true,
      "local_path": "papers/repair/d1fc0ae2f1fd2c4c6300.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1903.01611",
      "bytes": 1151102,
      "reused": false,
      "local_path": "papers/repair/c195d1bcc8a866b2436f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1906.02768",
      "bytes": 744366,
      "reused": false,
      "local_path": "papers/repair/9e6c7e60ac4f3cd07a7d.pdf"
    },
    {
      "status": "ok",
      "url": "https://papers.nips.cc/paper/8739-one-ticket-to-win-them-all-generalizing-lottery-ticket-initializations-across-datasets-and-optimizers.pdf?ref=ruder.io",
      "bytes": 322115,
      "reused": false,
      "local_path": "papers/repair/c3f2c00e461bbc1d087c.pdf"
    },
    {
      "status": "ok",
      "url": "https://papers.nips.cc/paper/8076-neural-tangent-kernel-convergence-and-generalization-in-neural-networks.pdf?ref=ruder.io",
      "bytes": 421992,
      "reused": false,
      "local_path": "papers/repair/6ab6a4b564a814fdb609.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1911.00809",
      "bytes": 553155,
      "reused": false,
      "local_path": "papers/repair/b6d31f00f9edbf775167.pdf"
    },
    {
      "status": "ok",
      "url": "https://papers.nips.cc/paper/9025-on-exact-computation-with-an-infinitely-wide-neural-net.pdf?ref=ruder.io",
      "bytes": 702293,
      "reused": false,
      "local_path": "papers/repair/c562f2ab7700431eaec4.pdf"
    },
    {
      "status": "ok",
      "url": "https://papers.nips.cc/paper/8928-cross-lingual-language-model-pretraining.pdf?ref=ruder.io",
      "bytes": 308680,
      "reused": false,
      "local_path": "papers/repair/f137de9479e99e14da7f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1911.02116",
      "bytes": 398981,
      "reused": true,
      "local_path": "papers/repair/069e86d8c4407636cd2f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1910.11856",
      "bytes": 614586,
      "reused": true,
      "local_path": "papers/repair/b6528b5b8ccdd3ecb04b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1911.01464",
      "bytes": 712704,
      "reused": true,
      "local_path": "papers/repair/018f96bf306b0f3f658f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.02450",
      "bytes": 7920816,
      "reused": false,
      "local_path": "papers/repair/338f9800ac3fb4c6e8aa.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1910.14599",
      "bytes": 1428177,
      "reused": false,
      "local_path": "papers/repair/00d5a582af8976ff5239.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1909.02487",
      "bytes": 6590042,
      "reused": false,
      "local_path": "papers/repair/6528b013c1e125376014.pdf"
    },
    {
      "status": "ok",
      "url": "http://papers.nips.cc/paper/9672-hamiltonian-neural-networks.pdf?ref=ruder.io",
      "bytes": 1871904,
      "reused": false,
      "local_path": "papers/repair/c121fb6993dc7f02207a.pdf"
    },
    {
      "status": "ok",
      "url": "https://www.aclweb.org/anthology/D19-1331.pdf?ref=ruder.io",
      "bytes": 334709,
      "reused": false,
      "local_path": "papers/repair/1d8e45d74dfa91e5698c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1901.08149",
      "bytes": 1524814,
      "reused": false,
      "local_path": "papers/repair/963a1e8123ccbf218e30.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1903.12136",
      "bytes": 1788635,
      "reused": false,
      "local_path": "papers/repair/73628a615c38541264ec.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1910.01108",
      "bytes": 436085,
      "reused": true,
      "local_path": "papers/repair/9d327e2a83cac2e9902e.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 18404,
  "expected_body_images": 11,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/202398738caa/pages/source.html",
  "local_html": "attachments/202398738caa/pages/article.html",
  "body_base_path": "attachments/202398738caa/pages/",
  "body_markdown_path": "attachments/202398738caa/pages/article.md"
}