{ "source_file": "0930", "source_platform": "个人博客", "author": "Sebastian Ruder", "title_listed": "ML and NLP Research Highlights of 2020", "keywords": "研究回顾,ML,NLP", "domain": "机器学习", "content_type": "行业分析", "is_paid_flag": "否", "url": "https://www.ruder.io/research-highlights-2020/", "publish_date": "2021-01", "corpus_stage": "预训练/SFT", "notes": "2020年ML与NLP研究亮点回顾", "id": "5f98aee439e5", "collected_at": "2026-10-02T14:16:16", "title": "ML and NLP Research Highlights of 2020", "body_markdown": "transfer learning\nML and NLP Research Highlights of 2020\nThis post summarizes progress in 10 exciting and impactful directions in ML and NLP in 2020.\nSebastian Ruder\n19 Jan 2021\n•\n15 min read\nShare\nThe selection of areas and methods is heavily influenced by my own interests; the selected topics are biased towards representation and transfer learning and towards natural language processing (NLP). I tried to cover the papers that I was aware of but likely missed many relevant ones—feel free to highlight them in the comments below. In all, I discuss the following highlights:\nScaling up—and down\nRetrieval augmentation\nFew-shot learning\nContrastive learning\nEvaluation beyond accuracy\nPractical concerns of large LMs\nMultilinguality\nImage Transformers\nML for science\nReinforcement learning\n1) Scaling up—and down\nModel sizes of language models from 2018–2020 (Credit:\nState of AI Report 2020\n)\nWhat happened?\n2020 saw the development of ever larger language and dialogue models such as Meena (\nAdiwardana et al., 2020\n),\nTuring-NLG\n, BST (\nRoller et al., 2020\n), and GPT-3 (\nBrown et al., 2020\n). At the same time, researchers have become more aware of how expensive and energy-hungry these models can be (\nStrubell et al., 2019\n) and work that focuses on making them smaller has gained momentum: Recent approaches rely on pruning (\nSajjad et al., 2020\n;\nFan et al., 2020a\n;\nSanh et al., 2020\n), quantization (\nFan et al., 2020b\n), distillation (\nSanh et al., 2019\n;\nSun et al., 2020\n), and compression (\nXu et al., 2020\n). Other approaches focused on making the Transformer architecture itself more efficient. Models in this line include the Performer (\nChoromanski et al., 2020\n) and Big Bird (\nZaheer et al., 2020\n), which can be seen in the cover image above. The image shows performance (y axis), speed (x axis) and memory footprint (circle size) of different models on the Long Range Arena benchmark (\nTay et al., 2020\n).\nTools such as the experiment-impact-tracker (\nHenderson et al., 2020\n) have made it easier to track the energy efficiency of models. They have also facilitated competitions and benchmarks that evaluate models primarily based on their efficiency such as the\nSustaiNLP workshop\nat EMNLP 2020, the\nEfficient QA competition\nat NeurIPS 2020, and HULK (\nZhou et al., 2020\n).\nWhy is it important?\nScaling up models allows us to keep pushing the boundaries of what current models can do. In order to deploy and use them in real-world scenarios, however, they need to be efficient. Ultimately, both directions benefit each other: Compressing large models yields efficient models with strong performance (\nLi et al., 2020\n) while more efficient methods may lead to stronger, larger models (\nClark et al., 2020\n).\nWhat's next?\nI am hopeful that—in light of the increasing interest in efficiency and the availability of tools—it will become more common not only to report a model's performance and number of parameters but also its energy efficiency. This should contribute to a more holistic evaluation that may help to bridge the gap to real-world ML use cases.\n2) Retrieval augmentation\nUnsupervised pre-training with REALM (\nGuu et al., 2020\n); retriever and encoder are jointly pre-trained\nWhat happened?\nLarge models have been shown to have learned a surprising amount of world knowledge from their pre-training data, which allows them to reproduce facts (\nJiang et al., 2020\n) and answer questions even without access to external context (\nRoberts et al., 2020\n). However, storing such knowledge implicitly in the parameters of a model is inefficient and requires ever larger models to retain more information. Instead, recent approaches jointly trained retrieval models and large language models, which led to strong results on knowledge-intensive NLP tasks such as open-domain question answering (\nGuu et al., 2020\n;\nLewis et al., 2020\n) and language modelling (\nKhandelwal et al., 2020\n). The main advantage of these methods is that they integrate retrieval directly into language model pre-training, which allows language models to be much more efficient by being able to off-load the recall of facts and focus on learning the more challenging aspects of natural language understanding. Consequently, the best systems in the NeurIPS 2020 EfficientQA competition (\nMin et al., 2020\n) all relied on retrieval.\nWhy is it important?\nRetrieval was the standard in many generative tasks, such as text summarization or dialogue and has largely been superseded by abstractive generation (\nAllahyari et al., 2017\n). Retrieval-augmented generation enables combining the best of both worlds: the factual correctness and faithfulness of retrieved segments and the relevancy and composition of generated text.\nWhat's next?\nRetrieval-augmented generation should be particularly useful for dealing with failure cases that have plagued generative neural models in the past, such as dealing with hallucinations (\nNie et al., 2019\n). It may also help make systems more interpretable by directly providing evidence for their prediction.\n3) Few-shot learning\nPrompt-based fine-tuning uses templated prompts and demonstrations (\nGao et al., 2020\n)\nWhat happened?\nOver the last years, driven by advances in pre-training, the number of training examples to perform a given task has progressively gone down (\nPeters et al., 2018\n;\nHoward et al., 2018\n). We are now at a stage where tens of examples can be used to demonstrate a given task (\nBansal et al., 2020\n). A very natural paradigm for few-shot learning is to reframe a task as language modelling. The most prominent instantiation of this, the in-context learning approach of GPT-3 (\nBrown et al., 2020\n) performs a prediction based on a few demonstrations of input–output pairs in the model's context and a prompt without any gradient updates. This setting, however, has a few limitations: It requires a huge model—without any updates the model needs to rely on its existing knowledge—, the amount of knowledge that the model can use is restricted by its context window, and prompts need to be hand-engineered.\nRecent work has sought to make such few-shot learning more effective by using a smaller model, integrating fine-tuning, and automatically generating natural language prompts (\nSchick and Schütze, 2020\n;\nGao et al., 2020\n;\nShin et al., 2020\n). Such work is closely related to the broader area of controllable neural text generation, which broadly seeks to leverage the generative capabilities of powerful pre-trained models. For an excellent overview, check out\nLilian Weng's blog post\n.\nFew-shot learning enables rapid adaptation of a model to many tasks. However, updating all model parameters for each task is wasteful. Instead, it is preferable to perform localized updates that concentrate changes in a small set of parameters. There have been a few approaches that make such efficient fine-tuning more practical including using adapters (\nHoulsby et al., 2019\n;\nPfeiffer et al., 2020a\n,\nb\n;\nÜstün et al., 2020\n), adding a sparse parameter vector (\nGuo et al., 2020\n), and only modifying bias values (\nBen-Zaken et al., 2020\n).\nWhy is it important?\nBeing able to teach a model a task based on only a few examples greatly reduces the barrier to entry for applying ML and NLP models in practice. This opens up applications where data is very expensive to collect and enables adapting models swiftly to new domains.\nWhat's next?\nFor many real-world scenarios, it is possible to collect thousands of training examples. Models should thus be able to scale seamlessly from learning from a few to learning from thousands of examples and should not be limited by e.g. their context length. Given that models have achieved super-human performance on many popular tasks such as\nSuperGLUE\nwhen fine-tuned on entire training datasets, enhancing their few-shot performance is a natural area for improvement.\n4) Contrastive learning\nInstance discrimination compares features from different transformations of the same images to each other (\nCaron et al., 2020\n)\nWhat happened?\nContrastive learning—learning to differentiate a positive example from negative samples, often from a noise distribution—such as using negative sampling or noise contrastive estimation is a staple of representation learning and self-supervised learning and a prominent part of classic approaches such as word2vec (\nMikolov et al., 2013\n). More recently, contrastive learning gained popularity in self-supervised representation learning in computer vision and speech (\nvan den Oord, 2018\n;\nHénaff et al., 2019\n). The recent generation of increasingly powerful self-supervised approaches for visual representation learning rely on contrastive learning using an instance discrimination task: different images are treated as negative pairs and views of the same image are treated as positive pairs. Recent approaches have further refined this general framework: SimCLR (\nChen et al., 2020\n) defines the contrastive loss over augmented examples, Momentum Contrast (\nHe et al., 2020\n) seeks to ensure a large and consistent set of pairs, SwAV (\nCaron et al., 2020\n) leverages online clustering, and BYOL only employs positive pairs (\nGrill et al., 2020\n).\nChen and He (2020\n) have furthermore proposed a simpler formulation that relates to the previous methods.\nRecently,\nZhao et al. (2020)\nfind that data augmentation is essential for contrastive learning. This might indicate why\nunsupervised\ncontrastive learning has not been successful with large pre-trained models in NLP where data augmentation is less common. They also hypothesize that the reason instance discrimination may work better than supervised pre-training in computer vision is that it does not try to make the features of all instances from a class similar but retains the information from each instance. This is less of a problem in NLP where unsupervised pre-training involves classification over thousands of word types. In NLP,\nGunel et al. (2020)\nrecently employ contrastive learning for\nsupervised\nfine-tuning.\nWhy is it important?\nThe cross-entropy objective between one-hot labels and a model's output logits commonly used in language modelling has several limitations such as generalizing poorly to imbalanced classes (\nCao et al., 2019\n). Contrastive learning is an alternative, complementary paradigm that may help ameliorate some of these deficits.\nWhat's next?\nContrastive learning combined with masked language modelling may enable us to learn representations that are richer and more robust. It could help model outliers and rare syntactic and semantic phenomena, which are a challenge for current NLP models.\n5) Evaluation beyond accuracy\nA CheckList template and tests probing for an understanding of negation in sentiment analysis (\nRibeiro et al., 2020\n)\nWhat happened?\nState-of-the-art models in NLP have achieved superhuman performance across many tasks. Whether or not we believe that such models can achieve true natural language understanding (\nYogatama et al., 2019\n;\nBender and Koller, 2020\n), we know that current models are not close to this elusive goal. However, the simple performance metrics of our tasks fail to capture the limitations of existing models. There are two key themes in this area: a) curating examples that are difficult for current models; and b) going beyond simple metrics such as accuracy towards more fine-grained evaluation.\nRegarding the former, the common methodology is to use adversarial filtering (\nZellers et al., 2018\n) during dataset creation to filter out examples that are predicted correctly by current models. Recent work proposes more efficient adversarial filtering methods (\nSakaguchi et al., 2020\n;\nLe Bras et al., 2020\n) and an iterative dataset creation process (\nNie et al., 2020\n;\nBartolo et al., 2020\n) where examples are filtered and models are re-trained over multiple rounds. A subset of such evolving benchmarks are available in\nDynabench\n.\nThe methods that regard the second point are similar in spirit. However, rather than creating examples that target a specific model, examples are used to probe for phenomena common to a task of interest. Commonly, minimal pairs—also known as counterfactual examples or contrast sets—(\nKaushik et al., 2020\n;\nGardner et al., 2020\n;\nWarstadt et al., 2020\n) are created, which perturb examples in a minimal way and often change the gold label.\nRibeiro et al. (2020)\nformalized some of the underlying intuitions in their CheckList framework, which enables the semi-automatic creation of such test cases. Alternatively, examples can be characterized based on different attributes, which allow a more fine-grained analysis of a model's strengths and weaknesses (\nFu et al., 2020\n).\nWhy is it important?\nIn order to make meaningful progress towards building more capable models machine learning models, we need to understand not only if a model outperforms a previous system but what kind of errors it makes and which phenomena it fails to capture.\nWhat's next?\nBy providing fine-grained diagnostics of model behaviour, it will be easier to identify a model's deficiencies and propose improvements that address them. Similarly, a fine-grained evaluation would allow a more nuanced comparison of the strengths and weaknesses of different methods.\n6) Practical concerns of large LMs\nModels generate toxic content based on seemingly innocuous prompts (\nGehman et al., 2020\n)\nWhat happened?\nCompared to 2019\nwhere the analysis of language models (LMs) mainly focused on the syntactic, semantic, and world knowledge that such models capture—see\n(Rogers et al., 2020)\nfor a great overview—recent analyses revealed a number of practical concerns. Pre-trained language models were found to be prone to generating toxic language (\nGehman et al., 2020\n) and leak information (\nSong & Raghunathan, 2020\n), to be susceptible to backdoors after fine-tuning, which let an attacker manipulate the model prediction (\nKurita et al., 2020\n;\nWallace et al., 2020\n), and to be vulnerable to model and data extraction attacks (\nKrishna et al., 2020\n;\nCarlini et al., 2020\n). In addition, pre-trained models are well known to capture biases with regard to protected attributes such as gender (\nBolukbasi et al., 2016\n;\nWebster et al., 2020\n)—see\n(Sun et al., 2019)\nfor an excellent survey on mitigating gender bias.\nWhy is it important?\nLarge pre-trained models are trained by many institutions and are actively deployed in real-world scenarios. It is thus of practical importance that we are not only aware of their biases but what behaviour may have actually harmful consequences.\nWhat's next?\nAs larger and more powerful models are developed, it is important that such practical concerns as well as issues around bias and fairness are part of the development process from the start.\n7) Multilinguality\nThe unequal distribution of labeled and unlabeled data for languages around the world (\nJoshi et al., 2020\n)\nWhat happened?\n2020 had many highlights in multilingual NLP. The\nMasakhane\norganisation whose mission is to strengthen NLP for African languages gave the\nkeynote\nat the\nFifth Conference on Machine Translation (WMT20)\n, one of the most inspiring presentations of the last year. New general-purpose benchmarks for other languages emerged including XTREME (\nHu et al., 2020\n), XGLUE (\nLiang et al., 2020\n), IndoNLU (\nWilie et al., 2020\n), IndicGLUE (\nKakwani et al., 2020\n). Existing datasets that were replicated in other languages—together with their non-English variants—include:\nSQuAD: XQuAD (\nArtetxe et al., 2020\n), MLQA (\nLewis et al., 2020\n), FQuAD (\nd'Hoffschmidt et al., 2020\n);\nNatural Questions: TyDiQA (\nClark et al., 2020\n), MKQA (\nLongpre et al., 2020\n);\nMNLI: OCNLI (\nHu et al., 2020\n), FarsTail (\nAmirkhani et al., 2020\n);\nthe CoNLL-09 dataset: X-SRL (\nDaza and Frank, 2020\n); and\nthe CNN/Daily Mail dataset: MLSUM (\nScialom et al., 2020\n).\nMany of these datasets and many others in different languages are easily accessible via\nHugging Face datasets\n. Powerful multilingual models that cover around 100 languages emerged including XML-R (\nConneau et al., 2020\n), RemBERT (\nChung et al., 2020\n), InfoXLM (\nChi et al., 2020\n), and others (see the\nXTREME leaderboard\nfor an overview). A plethora of language-specific BERT models have been trained for languages beyond English such as AraBERT (\nAntoun et al., 2020\n) and IndoBERT (\nWilie et al., 2020\n); see (\nNozza et al., 2020\n;\nRust et al., 2020\n) for an overview. With efficient multilingual frameworks such as\nAdapterHub\n(\nPfeiffer et al., 2020\n),\nStanza\n(\nQi et al., 2020\n) and\nTrankit\n(\nNguyen et al., 2020\n) it has become easier than ever to apply and build models for many of the world's languages.\nFinally, two position papers that inspired much of my thinking in this area this year are\nThe State and Fate of Linguistic Diversity and Inclusion in the NLP World\n(Joshi et al., 2020)\nand\nDecolonising Speech and Language Technology\n(\nBird, 2020\n). While the first highlights the urgent importance of working on languages beyond English, the second one cautions against treating language communities and their data as a commodity.\nWhy is it important?\nWorking on NLP beyond English\nhas numerous benefits\n: It poses interesting challenges for ML and NLP and enables having a large impact on society, among many others.\nWhat's next?\nGiven the availability of data and models in different languages, the stage is set to make meaningful progress on languages beyond English. I am most excited about developing models that tackle the most challenging settings and identifying in which cases the assumptions that underlie our current models fail.\n8) Image Transformers\nThe Vision Transformer (\nDosovitskiy et al., 2020\n) applies a Transformer encoder to flattened image patches\nWhat happened?\nWhile Transformers have achieved large success in NLP, they were—up until recently—less successful in computer vision where convolutional neural networks (CNNs) still reigned supreme. While models early in the year such as DETR (\nCarion et al., 2020\n) employed a CNN to compute image features, later models were completely convolution-free. Image GPT (\nChen et al., 2020\n) applied the GPT-2 recipe to pre-training directly from pixels and outperforms a supervised Wide ResNet. Later models all reshape an image into patches that are treated as \"tokens\". Vision Transformer (\nDosovitskiy et al., 2020\n) is pre-trained on millions of labelled images—each consisting of such patches—outperforming state-of-the-art CNNs. The Image Processing Transformer (\nChen et al., 2020\n) pre-trains on corrupted ImageNet examples with a contrastive loss and achieves state-of-the-art performance on low-level image tasks. The Data-efficient image Transformer (\nTouvron et al., 2020\n) is pre-trained on ImageNet via distillation. Interestingly, they observe that CNNs are better teachers. This is similar to findings for distilling an inductive bias into BERT (\nKuncoro et al., 2020\n). In contrast in speech, Transformers have not been applied directly to the audio signal—to my knowledge—but typically receive the output of an encoder such as a CNN as input (\nMoritz et al., 2020\n;\nGulati et al., 2020\n;\nConneau et al., 2020\n)\nWhy is it important?\nTransformers have less inductive bias compared to CNNs and RNNs. While being less theoretically powerful than RNNs (\nWeiss et al., 2018\n;\nHahn et al., 2020\n), given sufficient data and scale Transformers have been shown to eventually outperform their inductively biased competitors (cf.\nThe Bitter Lesson\n).\nWhat's next?\nWe will likely see Transformers become more popular in computer vision. They will be applied particularly in scenarios where enough compute and data for unsupervised pre-training is available. In smaller scale settings, CNNs will likely still be the go-to approach and a strong baseline.\n9) ML for science\nThe self-attention-based architecture of AlphaFold (Credit:\nDeepMind blog\n)\nWhat happened?\nOne of the highlights was\nAlphaFold\ndemonstrating ground-breaking performance in the biannual CASP challenge for protein folding. Beyond that, there have been several other notable developments in applying ML to problems in the natural sciences. MetNet (\nSønderby et al., 2020\n) outperformed numerical weather prediction for precipitation forecasting,\nLample and Charton (2020)\nsolved differential equations using neural networks better than commercial computer algebra systems, and\nBellemare et al. (2020)\nused reinforcement learning to navigate balloons in the stratosphere.\nIn addition, ML has been used extensively to help with the ongoing COVID-19 pandemic, e.g. to forecast COVID-19 spread (\nKapoor et al., 2020\n),\npredict structures associated with COVID-19\n, translate relevant data into 35 different languages (\nAnastasopoulos et al., 2020\n), and answer questions about COVID-19 in real-time (\nLee et al., 2020\n). For an overview of COVID-19 related applications of NLP, check out the\nProceedings of the 1st Workshop on NLP for COVID-19\n.\nWhy is it important?\nThe natural sciences are arguably the most impactful application area for ML. Improvements touch many aspects of life and can have a profound impact on the world.\nWhat's next?\nWith progress in areas as central as protein folding, the speed of application of ML to the natural sciences will only accelerate. I am looking forward to many more fundamental advances that have a positive impact in the world.\n10) Reinforcement learning\nPerformance of Agent57 and MuZero on Atari compared to state-of-the-art agents in terms of the number of games where they outperform the human benchmark throughout training (\nBadia et al., 2020\n)\nWhat happened?\nFor the first time, a single deep RL agent—Agent57 (\nBadia et al., 2020\n)—has achieved superhuman performance on all 57 Atari games, a long-standing benchmark in the deep reinforcement learning literature. The agent's versatility comes from a neural network that allows it to switch between exploratory and exploitative policies. Another milestone was the development of MuZero (\nSchrittwieser et al., 2020\n), which predicts the aspects of the environment that are most important for accurate planning. Without any knowledge of the game dynamics, it achieved state-of-the-art performance on Atari as well as superhuman performance on Go, chess, and shogi. Finally, Munchausen RL agents (\nVieillard et al., 2020\n) improved on state-of-the-art agents via a simple, theoretically founded modification.\nWhy is it important?\nReinforcement learning algorithms have a multitude of practical implications (\nBellemare et al., 2020\n). Improvements over the fundamental algorithms in this area can have a large practical impact by enabling better planning, environment modelling, and action prediction.\nWhat's next?\nWith classic benchmarks such as Atari essentially solved, researchers may look to more challenging settings to test their algorithms such as generalizing to out-of-distribution tasks, improving sample-efficiency, multi-task learning, etc.\nCitation\nFor attribution in academic contexts, please cite this work as:\n@misc{ruder2021researchhighlights,\n author = {Ruder, Sebastian},\n title = {{ML and NLP Research Highlights of 2020}},\n year = {2021},\n howpublished = {\\url{http://ruder.io/research-highlights-2020}},\n}\nThanks to Sameer Singh whose\nTwitter thread\nreviewing NLP research in 2020 provided inspiration for this post.", "metadata": { "title": "ML and NLP Research Highlights of 2020", "description": "This post summarizes progress in 10 exciting and impactful directions in ML and NLP in 2020.", "keywords": null, "author": null, "og:site_name": "ruder.io", "og:type": "article", "og:title": "ML and NLP Research Highlights of 2020", "og:description": "This post summarizes progress in 10 exciting and impactful directions in ML and NLP in 2020.", "og:url": "https://www.ruder.io/research-highlights-2020/", "og:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/lra_analysis-2.png", "og:image:width": "1125", "og:image:height": "487", "twitter:card": "summary_large_image", "twitter:title": "ML and NLP Research Highlights of 2020", "twitter:description": "This post summarizes progress in 10 exciting and impactful directions in ML and NLP in 2020.", "twitter:url": "https://www.ruder.io/research-highlights-2020/", "twitter:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/lra_analysis-2.png", "twitter:label1": "Written by", "twitter:data1": "Sebastian Ruder", "twitter:label2": "Filed under", "twitter:data2": "transfer learning, natural language processing, language models, reinforcement learning", "twitter:site": "@seb_ruder", "article:published_time": "2021-01-19T09:00:00.000Z", "article:modified_time": "2022-02-06T12:29:11.000Z", "article:tag": "reinforcement learning" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/5f98aee439e5/code/body-000.txt", "chars": 205, "language": "text" } ], "images": [ { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2021/01/lra_analysis-2.png", "bytes": 73962, "reused": true, "local_path": "attachments/5f98aee439e5/images/001.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2021/01/lra_analysis-2.png", "alt": "ML and NLP Research Highlights of 2020", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/large_models.png", "bytes": 145628, "reused": true, "local_path": "attachments/5f98aee439e5/images/005.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/large_models.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/retrieval_augmented_lm.png", "bytes": 40621, "reused": true, "local_path": "attachments/5f98aee439e5/images/006.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/retrieval_augmented_lm.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/prompt-based_fine-tuning.png", "bytes": 47685, "reused": true, "local_path": "attachments/5f98aee439e5/images/007.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/prompt-based_fine-tuning.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/instance_discrimination.png", "bytes": 17935, "reused": false, "local_path": "attachments/5f98aee439e5/images/body-216c4695da0604ee3ec0.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/instance_discrimination.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/checklist_negation.png", "bytes": 34120, "reused": false, "local_path": "attachments/5f98aee439e5/images/body-196c5a3c28f17b8d4580.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/checklist_negation.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/real_toxicity_prompts.png", "bytes": 182869, "reused": false, "local_path": "attachments/5f98aee439e5/images/body-8a2ae8a61dbe32ebcffe.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/real_toxicity_prompts.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/language_resource_distribution.png", "bytes": 356366, "reused": false, "local_path": "attachments/5f98aee439e5/images/body-44207090a05a79ba0292.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/language_resource_distribution.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/vision_transformer.png", "bytes": 409689, "reused": false, "local_path": "attachments/5f98aee439e5/images/body-59c6418de9cdf6ecf432.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/vision_transformer.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/alphafold.png", "bytes": 571246, "reused": false, "local_path": "attachments/5f98aee439e5/images/body-c29107c7bcd831092fc7.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/alphafold.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/atari_performance.png", "bytes": 170638, "reused": false, "local_path": "attachments/5f98aee439e5/images/body-4169fc7399446eb87095.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/01/atari_performance.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/watch?v=Xbc_g_OknqA", "status": "ok", "local_path": "attachments/5f98aee439e5/videos/Xbc_g_OknqA.mp4", "bytes": 326929424, "valid": true, "height": 1080, "width": 1920, "duration": "3631.461000" } ], "links": [ { "href": "https://www.ruder.io/tag/transfer-learning/", "label": "transfer learning", "type": "a" }, { "href": "https://www.ruder.io/author/sebastian/", "label": "", "type": "a" }, { "href": "https://www.stateof.ai/?ref=ruder.io", "label": "State of AI Report 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2001.09977?ref=ruder.io", "label": "Adiwardana et al., 2020", "type": "a" }, { "href": "https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/?ref=ruder.io", "label": "Turing-NLG", "type": "a" }, { "href": "https://arxiv.org/abs/2004.13637?ref=ruder.io", "label": "Roller et al., 2020", "type": "a" }, { "href": "https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf?ref=ruder.io", "label": "Brown et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P19-1355/?ref=ruder.io", "label": "Strubell et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/2004.03844?ref=ruder.io", "label": "Sajjad et al., 2020", "type": "a" }, { "href": "https://openreview.net/forum?id=SylO2yStDr&ref=ruder.io", "label": "Fan et al., 2020a", "type": "a" }, { "href": "https://papers.nips.cc/paper/2020/file/eae15aabaa768ae4a5993a8a4f4fa6e4-Paper.pdf?ref=ruder.io", "label": "Sanh et al., 2020", "type": "a" }, { "href": "https://openreview.net/forum?id=dV19Yyi1fS3&ref=ruder.io", "label": "Fan et al., 2020b", "type": "a" }, { "href": "https://arxiv.org/abs/1910.01108?ref=ruder.io", "label": "Sanh et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.acl-main.195/?ref=ruder.io", "label": "Sun et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.emnlp-main.633/?ref=ruder.io", "label": "Xu et al., 2020", "type": "a" }, { "href": "https://openreview.net/forum?id=Ua6zuk0WRH&ref=ruder.io", "label": "Choromanski et al., 2020", "type": "a" }, { "href": "https://papers.nips.cc/paper/2020/file/c8512d142a2d849725f31a9a7a361ab9-Paper.pdf?ref=ruder.io", "label": "Zaheer et al., 2020", "type": "a" }, { "href": "https://openreview.net/forum?id=qVyeW-grC2k&ref=ruder.io", "label": "Tay et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2002.05651?ref=ruder.io", "label": "Henderson et al., 2020", "type": "a" }, { "href": "https://sites.google.com/view/sustainlp2020/home?utm_campaign=NLP%20News&utm_medium=email&utm_source=Revue%20newsletter", "label": "SustaiNLP workshop", "type": "a" }, { "href": "https://ai.google.com/research/NaturalQuestions/efficientqa?utm_campaign=NLP%20News&utm_medium=email&utm_source=Revue%20newsletter", "label": "Efficient QA competition", "type": "a" }, { "href": "https://arxiv.org/abs/2002.05829?ref=ruder.io", "label": "Zhou et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2002.11794?ref=ruder.io", "label": "Li et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2003.10555?ref=ruder.io", "label": "Clark et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2002.08909?ref=ruder.io", "label": "Guu et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.tacl-1.28.pdf?ref=ruder.io", "label": "Jiang et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.emnlp-main.437.pdf?ref=ruder.io", "label": "Roberts et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2005.11401?ref=ruder.io", "label": "Lewis et al., 2020", "type": "a" }, { "href": "https://openreview.net/forum?id=HklBjCEKvH&ref=ruder.io", "label": "Khandelwal et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2101.00133?ref=ruder.io", "label": "Min et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/1707.02268?ref=ruder.io", "label": "Allahyari et al., 2017", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P19-1256/?ref=ruder.io", "label": "Nie et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/2012.15723?ref=ruder.io", "label": "Gao et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/N18-1202/?ref=ruder.io", "label": "Peters et al., 2018", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P18-1031/?ref=ruder.io", "label": "Howard et al., 2018", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.emnlp-main.38/?ref=ruder.io", "label": "Bansal et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2009.07118?ref=ruder.io", "label": "Schick and Schütze, 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2010.15980?ref=ruder.io", "label": "Shin et al., 2020", "type": "a" }, { "href": "https://lilianweng.github.io/lil-log/2021/01/02/controllable-neural-text-generation.html?ref=ruder.io", "label": "Lilian Weng's blog post", "type": "a" }, { "href": "http://proceedings.mlr.press/v97/houlsby19a/houlsby19a.pdf?ref=ruder.io", "label": "Houlsby et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.emnlp-main.617/?ref=ruder.io", "label": "Pfeiffer et al., 2020a", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.emnlp-demos.7/?ref=ruder.io", "label": "b", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.emnlp-main.180.pdf?ref=ruder.io", "label": "Üstün et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2012.07463?ref=ruder.io", "label": "Guo et al., 2020", "type": "a" }, { "href": "https://nlp.biu.ac.il/~yogo/bitfit.pdf?ref=ruder.io", "label": "Ben-Zaken et al., 2020", "type": "a" }, { "href": "https://super.gluebenchmark.com/leaderboard?ref=ruder.io", "label": "SuperGLUE", "type": "a" }, { "href": "https://proceedings.neurips.cc/paper/2020/file/70feb62b69f16e0238f741fab228fec2-Paper.pdf?ref=ruder.io", "label": "Caron et al., 2020", "type": "a" }, { "href": "https://papers.nips.cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf?ref=ruder.io", "label": "Mikolov et al., 2013", "type": "a" }, { "href": "https://arxiv.org/abs/1807.03748?ref=ruder.io", "label": "van den Oord, 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1905.09272?ref=ruder.io", "label": "Hénaff et al., 2019", "type": "a" }, { "href": "http://proceedings.mlr.press/v119/chen20j/chen20j.pdf?ref=ruder.io", "label": "Chen et al., 2020", "type": "a" }, { "href": "https://openaccess.thecvf.com/content_CVPR_2020/papers/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_CVPR_2020_paper.pdf?ref=ruder.io", "label": "He et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2006.07733?ref=ruder.io", "label": "Grill et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2011.10566?ref=ruder.io", "label": "Chen and He (2020", "type": "a" }, { "href": "https://openreview.net/forum?id=tC6iW2UUbJf&ref=ruder.io", "label": "Zhao et al. (2020)", "type": "a" }, { "href": "https://openreview.net/forum?id=cu7IUiOhujH&ref=ruder.io", "label": "Gunel et al. (2020)", "type": "a" }, { "href": "http://papers.neurips.cc/paper/8435-learning-imbalanced-datasets-with-label-distribution-aware-margin-loss.pdf?ref=ruder.io", "label": "Cao et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.acl-main.442/?ref=ruder.io", "label": "Ribeiro et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/1901.11373?ref=ruder.io", "label": "Yogatama et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.acl-main.463/?ref=ruder.io", "label": "Bender and Koller, 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D18-1009/?ref=ruder.io", "label": "Zellers et al., 2018", "type": "a" }, { "href": "https://ojs.aaai.org/index.php/AAAI/article/view/6399/6255?ref=ruder.io", "label": "Sakaguchi et al., 2020", "type": "a" }, { "href": "http://proceedings.mlr.press/v119/bras20a/bras20a.pdf?ref=ruder.io", "label": "Le Bras et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.acl-main.441/?ref=ruder.io", "label": "Nie et al., 2020", "type": "a" }, { "href": "https://transacl.org/ojs/index.php/tacl/article/view/2129/649?ref=ruder.io", "label": "Bartolo et al., 2020", "type": "a" }, { "href": "https://dynabench.org/?ref=ruder.io", "label": "Dynabench", "type": "a" }, { "href": "https://openreview.net/forum?id=Sklgs0NFvr&ref=ruder.io", "label": "Kaushik et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2004.02709?ref=ruder.io", "label": "Gardner et al., 2020", "type": "a" }, { "href": "https://transacl.org/ojs/index.php/tacl/article/view/2013/527?ref=ruder.io", "label": "Warstadt et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.emnlp-main.489/?ref=ruder.io", "label": "Fu et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.findings-emnlp.301/?ref=ruder.io", "label": "Gehman et al., 2020", "type": "a" }, { "href": "https://ruder.io/research-highlights-2019/?ref=ruder.io", "label": "Compared to 2019", "type": "a" }, { "href": "https://www.mitpressjournals.org/doi/full/10.1162/tacl_a_00349?ref=ruder.io", "label": "(Rogers et al., 2020)", "type": "a" }, { "href": "https://arxiv.org/abs/2004.00053?ref=ruder.io", "label": "Song & Raghunathan, 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.acl-main.249/?ref=ruder.io", "label": "Kurita et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2010.12563?ref=ruder.io", "label": "Wallace et al., 2020", "type": "a" }, { "href": "https://openreview.net/forum?id=Byl5NREFDr&ref=ruder.io", "label": "Krishna et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2012.07805?ref=ruder.io", "label": "Carlini et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/1607.06520?ref=ruder.io", "label": "Bolukbasi et al., 2016", "type": "a" }, { "href": "https://arxiv.org/abs/2010.06032?ref=ruder.io", "label": "Webster et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P19-1159/?ref=ruder.io", "label": "(Sun et al., 2019)", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.acl-main.560/?ref=ruder.io", "label": "Joshi et al., 2020", "type": "a" }, { "href": "https://www.masakhane.io/?ref=ruder.io", "label": "Masakhane", "type": "a" }, { "href": "https://www.youtube.com/watch?v=Xbc_g_OknqA&ref=ruder.io", "label": "keynote", "type": "a" }, { "href": "http://www.statmt.org/wmt20/?ref=ruder.io", "label": "Fifth Conference on Machine Translation (WMT20)", "type": "a" }, { "href": "http://proceedings.mlr.press/v119/hu20b/hu20b.pdf?ref=ruder.io", "label": "Hu et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.emnlp-main.484/?ref=ruder.io", "label": "Liang et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.aacl-main.85/?ref=ruder.io", "label": "Wilie et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.findings-emnlp.445/?ref=ruder.io", "label": "Kakwani et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.acl-main.421/?ref=ruder.io", "label": "Artetxe et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.acl-main.653/?ref=ruder.io", "label": "Lewis et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2002.06071?ref=ruder.io", "label": "d'Hoffschmidt et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2003.05002?ref=ruder.io", "label": "Clark et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2007.15207?ref=ruder.io", "label": "Longpre et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.findings-emnlp.314/?ref=ruder.io", "label": "Hu et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2009.08820?ref=ruder.io", "label": "Amirkhani et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.emnlp-main.321/?ref=ruder.io", "label": "Daza and Frank, 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.emnlp-main.647/?ref=ruder.io", "label": "Scialom et al., 2020", "type": "a" }, { "href": "https://huggingface.co/datasets?ref=ruder.io", "label": "Hugging Face datasets", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.acl-main.747/?ref=ruder.io", "label": "Conneau et al., 2020", "type": "a" }, { "href": "https://openreview.net/forum?id=xpFFI_NtgpW&ref=ruder.io", "label": "Chung et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2007.07834?ref=ruder.io", "label": "Chi et al., 2020", "type": "a" }, { "href": "https://sites.research.google/xtreme?ref=ruder.io", "label": "XTREME leaderboard", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.osact-1.2/?ref=ruder.io", "label": "Antoun et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2003.02912?ref=ruder.io", "label": "Nozza et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2012.15613?ref=ruder.io", "label": "Rust et al., 2020", "type": "a" }, { "href": "https://adapterhub.ml/?ref=ruder.io", "label": "AdapterHub", "type": "a" }, { "href": "https://stanfordnlp.github.io/stanza/?ref=ruder.io", "label": "Stanza", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.acl-demos.14/?ref=ruder.io", "label": "Qi et al., 2020", "type": "a" }, { "href": "https://github.com/nlp-uoregon/trankit?ref=ruder.io", "label": "Trankit", "type": "a" }, { "href": "https://arxiv.org/abs/2101.03289?ref=ruder.io", "label": "Nguyen et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.coling-main.313/?ref=ruder.io", "label": "Bird, 2020", "type": "a" }, { "href": "https://ruder.io/nlp-beyond-english/?ref=ruder.io", "label": "has numerous benefits", "type": "a" }, { "href": "https://openreview.net/forum?id=YicbFdNTTy&ref=ruder.io", "label": "Dosovitskiy et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2005.12872?ref=ruder.io", "label": "Carion et al., 2020", "type": "a" }, { "href": "http://proceedings.mlr.press/v119/chen20s/chen20s.pdf?ref=ruder.io", "label": "Chen et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2012.00364?ref=ruder.io", "label": "Chen et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2012.12877?ref=ruder.io", "label": "Touvron et al., 2020", "type": "a" }, { "href": "https://www.mitpressjournals.org/doi/full/10.1162/tacl_a_00345?ref=ruder.io", "label": "Kuncoro et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2001.02674?ref=ruder.io", "label": "Moritz et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2005.08100?ref=ruder.io", "label": "Gulati et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2006.13979?ref=ruder.io", "label": "Conneau et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P18-2117/?ref=ruder.io", "label": "Weiss et al., 2018", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.tacl-1.11/?ref=ruder.io", "label": "Hahn et al., 2020", "type": "a" }, { "href": "http://www.incompleteideas.net/IncIdeas/BitterLesson.html?ref=ruder.io", "label": "The Bitter Lesson", "type": "a" }, { "href": "https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology?ref=ruder.io", "label": "DeepMind blog", "type": "a" }, { "href": "https://arxiv.org/abs/2003.12140?ref=ruder.io", "label": "Sønderby et al., 2020", "type": "a" }, { "href": "https://openreview.net/forum?id=S1eZYeHFDS&ref=ruder.io", "label": "Lample and Charton (2020)", "type": "a" }, { "href": "https://www.nature.com/articles/s41586-020-2939-8?ref=ruder.io", "label": "Bellemare et al. (2020)", "type": "a" }, { "href": "https://arxiv.org/abs/2007.03113?ref=ruder.io", "label": "Kapoor et al., 2020", "type": "a" }, { "href": "https://deepmind.com/research/open-source/computational-predictions-of-protein-structures-associated-with-COVID-19?ref=ruder.io", "label": "predict structures associated with COVID-19", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.nlpcovid19-2.5/?ref=ruder.io", "label": "Anastasopoulos et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/2020.nlpcovid19-2.1/?ref=ruder.io", "label": "Lee et al., 2020", "type": "a" }, { "href": "https://www.aclweb.org/anthology/volumes/2020.nlpcovid19-2/?ref=ruder.io", "label": "Proceedings of the 1st Workshop on NLP for COVID-19", "type": "a" }, { "href": "https://arxiv.org/abs/2003.13350?ref=ruder.io", "label": "Badia et al., 2020", "type": "a" }, { "href": "https://www.nature.com/articles/s41586-020-03051-4?ref=ruder.io", "label": "Schrittwieser et al., 2020", "type": "a" }, { "href": "https://papers.nips.cc/paper/2020/file/2c6a0bae0f071cbbf0bb3d5b11d90a82-Supplemental.pdf?ref=ruder.io", "label": "Vieillard et al., 2020", "type": "a" }, { "href": "https://twitter.com/sameer_/status/1347472491346763776?ref=ruder.io", "label": "Twitter thread", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:52:02Z", "source_url": "https://www.ruder.io/research-highlights-2020/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/2001.09977", "bytes": 1035517, "reused": false, "local_path": "papers/repair/21ef562c324f2c690df1.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2004.13637", "bytes": 3481988, "reused": false, "local_path": "papers/repair/587400c5f89d7e6b8fc9.pdf" }, { "status": "ok", "url": "https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf?ref=ruder.io", "bytes": 1283694, "reused": false, "local_path": "papers/repair/7496c18caa2b891415a8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2004.03844", "bytes": 1220038, "reused": false, "local_path": "papers/repair/2199ffb88d795d5991c3.pdf" }, { "status": "ok", "url": "https://papers.nips.cc/paper/2020/file/eae15aabaa768ae4a5993a8a4f4fa6e4-Paper.pdf?ref=ruder.io", "bytes": 2040321, "reused": true, "local_path": "papers/repair/356203d70bc75b162977.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1910.01108", "bytes": 436085, "reused": true, "local_path": "papers/repair/9d327e2a83cac2e9902e.pdf" }, { "status": "ok", "url": "https://papers.nips.cc/paper/2020/file/c8512d142a2d849725f31a9a7a361ab9-Paper.pdf?ref=ruder.io", "bytes": 639025, "reused": false, "local_path": "papers/repair/dc7feeb92d017a44c341.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2002.05651", "bytes": 2128835, "reused": false, "local_path": "papers/repair/4b82d1a56e5e069fb486.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2002.05829", "bytes": 334309, "reused": false, "local_path": "papers/repair/de2af81000d4a7230bfc.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2002.11794", "bytes": 1743499, "reused": false, "local_path": "papers/repair/1e75227936c8367d8a73.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2003.10555", "bytes": 497999, "reused": false, "local_path": "papers/repair/95df7a9ffe51e8dc3c44.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2002.08909", "bytes": 419037, "reused": true, "local_path": "papers/repair/9325ba1ac58281154385.pdf" }, { "status": "ok", "url": "https://www.aclweb.org/anthology/2020.tacl-1.28.pdf?ref=ruder.io", "bytes": 461737, "reused": false, "local_path": "papers/repair/588b754482fb4770db51.pdf" }, { "status": "ok", "url": "https://www.aclweb.org/anthology/2020.emnlp-main.437.pdf?ref=ruder.io", "bytes": 332899, "reused": false, "local_path": "papers/repair/e69da3f14864dfb2e05c.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2005.11401", "bytes": 885323, "reused": true, "local_path": "papers/repair/fc8d32f196e0f0c27ffb.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2101.00133", "bytes": 2904096, "reused": true, "local_path": "papers/repair/a3139cdfd88862860aa5.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1707.02268", "bytes": 231818, "reused": false, "local_path": "papers/repair/72a63b576f8494f85cb2.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2012.15723", "bytes": 1441484, "reused": true, "local_path": "papers/repair/f9ff70b06a4d3ed4bae9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2009.07118", "bytes": 468372, "reused": true, "local_path": "papers/repair/e950ca6026acc3b4cc56.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2010.15980", "bytes": 663180, "reused": true, "local_path": "papers/repair/5fc241cafafb7eeca9db.pdf" }, { "status": "ok", "url": "http://proceedings.mlr.press/v97/houlsby19a/houlsby19a.pdf?ref=ruder.io", "bytes": 724447, "reused": true, "local_path": "papers/repair/d30e94c6755c97464157.pdf" }, { "status": "ok", "url": "https://www.aclweb.org/anthology/2020.emnlp-main.180.pdf?ref=ruder.io", "bytes": 2810367, "reused": false, "local_path": "papers/repair/b474d4e5d1b3834a2238.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2012.07463", "bytes": 513545, "reused": true, "local_path": "papers/repair/d63745904b0f32bb43c6.pdf" }, { "status": "ok", "url": "https://nlp.biu.ac.il/~yogo/bitfit.pdf?ref=ruder.io", "bytes": 187901, "reused": true, "local_path": "papers/repair/14c2cd6cd14cd2f14157.pdf" }, { "status": "ok", "url": "https://proceedings.neurips.cc/paper/2020/file/70feb62b69f16e0238f741fab228fec2-Paper.pdf?ref=ruder.io", "bytes": 382271, "reused": false, "local_path": "papers/repair/0f7ad45392f3113e2631.pdf" }, { "status": "ok", "url": "https://papers.nips.cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf?ref=ruder.io", "bytes": 111997, "reused": false, "local_path": "papers/repair/696f67590115667c5340.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1807.03748", "bytes": 3182089, "reused": true, "local_path": "papers/repair/7ace1819f0b552bee2ab.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.09272", "bytes": 2411659, "reused": true, "local_path": "papers/repair/b9006248b176b4b194f1.pdf" }, { "status": "ok", "url": "http://proceedings.mlr.press/v119/chen20j/chen20j.pdf?ref=ruder.io", "bytes": 3328680, "reused": false, "local_path": "papers/repair/41f50fdfe69012f3adf8.pdf" }, { "status": "ok", "url": "https://openaccess.thecvf.com/content_CVPR_2020/papers/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_CVPR_2020_paper.pdf?ref=ruder.io", "bytes": 497030, "reused": false, "local_path": "papers/repair/e8cfe07d6dbc765dfab5.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2006.07733", "bytes": 4837213, "reused": true, "local_path": "papers/repair/7c03b7b76959caf6d3ab.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2011.10566", "bytes": 579388, "reused": false, "local_path": "papers/repair/4daf5d47207a3ce45f6e.pdf" }, { "status": "ok", "url": "http://papers.neurips.cc/paper/8435-learning-imbalanced-datasets-with-label-distribution-aware-margin-loss.pdf?ref=ruder.io", "bytes": 398351, "reused": false, "local_path": "papers/repair/9641a34f0bfe51bb7156.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1901.11373", "bytes": 1034653, "reused": false, "local_path": "papers/repair/a65e297158fc2c17bcf7.pdf" }, { "status": "ok", "url": "http://proceedings.mlr.press/v119/bras20a/bras20a.pdf?ref=ruder.io", "bytes": 1153152, "reused": false, "local_path": "papers/repair/ddc03355590135266155.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2004.02709", "bytes": 2397965, "reused": false, "local_path": "papers/repair/b76d0e7d7504e0262370.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2004.00053", "bytes": 852273, "reused": false, "local_path": "papers/repair/ad2ae59a3e89a7f4b6b4.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2010.12563", "bytes": 923298, "reused": false, "local_path": "papers/repair/f9325f5da25c0f21d027.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2012.07805", "bytes": 3229447, "reused": true, "local_path": "papers/repair/a707ec064e66b4a9f697.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1607.06520", "bytes": 738680, "reused": false, "local_path": "papers/repair/a02da4a6ebe1ff5fc25e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2010.06032", "bytes": 453220, "reused": false, "local_path": "papers/repair/1f0c4b34007124280516.pdf" }, { "status": "ok", "url": "http://proceedings.mlr.press/v119/hu20b/hu20b.pdf?ref=ruder.io", "bytes": 513254, "reused": false, "local_path": "papers/repair/43ec68a7c76c2de815d9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2002.06071", "bytes": 708654, "reused": false, "local_path": "papers/repair/8ba8c4149ca5b0656e6b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2003.05002", "bytes": 171338, "reused": false, "local_path": "papers/repair/d90354cce125264be057.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2007.15207", "bytes": 1351038, "reused": false, "local_path": "papers/repair/e267de7fa087c8c40cd1.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2009.08820", "bytes": 678934, "reused": false, "local_path": "papers/repair/b41c01061e6dead7843b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2007.07834", "bytes": 400104, "reused": false, "local_path": "papers/repair/30f3c650eebdb714e643.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2003.02912", "bytes": 285308, "reused": false, "local_path": "papers/repair/02dd9c4f9b6003d99811.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2012.15613", "bytes": 847430, "reused": false, "local_path": "papers/repair/44f1276778c54c236445.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2101.03289", "bytes": 727927, "reused": false, "local_path": "papers/repair/5557cc9a2407a624b978.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2005.12872", "bytes": 7634041, "reused": false, "local_path": "papers/repair/927d7164ed07f134c104.pdf" }, { "status": "ok", "url": "http://proceedings.mlr.press/v119/chen20s/chen20s.pdf?ref=ruder.io", "bytes": 2093643, "reused": false, "local_path": "papers/repair/4682d14ee78d36e74143.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2012.00364", "bytes": 15490668, "reused": false, "local_path": "papers/repair/eab6216a41e640d0f234.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2012.12877", "bytes": 558744, "reused": false, "local_path": "papers/repair/485ef56912152afee938.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2001.02674", "bytes": 615524, "reused": false, "local_path": "papers/repair/b57162a05b4740d91175.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2005.08100", "bytes": 412099, "reused": false, "local_path": "papers/repair/0d5e0b5699a5364792fd.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2006.13979", "bytes": 487701, "reused": false, "local_path": "papers/repair/1a028f72da584b12f75d.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2003.12140", "bytes": 5635202, "reused": false, "local_path": "papers/repair/921b59a89858dcf3f044.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2007.03113", "bytes": 2442024, "reused": false, "local_path": "papers/repair/1cd3058bab5d9bc5890b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2003.13350", "bytes": 4229515, "reused": true, "local_path": "papers/repair/e9c227fe1baabe459214.pdf" }, { "status": "ok", "url": "https://papers.nips.cc/paper/2020/file/2c6a0bae0f071cbbf0bb3d5b11d90a82-Supplemental.pdf?ref=ruder.io", "bytes": 11772383, "reused": false, "local_path": "papers/repair/48506f7c4aada3808aa4.pdf" } ], "actual_video_files": [ { "local_path": "attachments/5f98aee439e5/videos/Xbc_g_OknqA.mp4", "bytes": 326929424, "valid": true, "height": 1080, "width": 1920, "duration": "3631.461000", "status": "ok" } ], "body_chars": 23358, "expected_body_images": 11, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/5f98aee439e5/pages/source.html", "local_html": "attachments/5f98aee439e5/pages/article.html", "body_base_path": "attachments/5f98aee439e5/pages/", "body_markdown_path": "attachments/5f98aee439e5/pages/article.md" }