{ "source_file": "0930", "source_platform": "个人博客", "author": "Sebastian Ruder", "title_listed": "Challenges and Opportunities in NLP Benchmarking", "keywords": "基准,评估,NLP", "domain": "文本处理", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://www.ruder.io/nlp-benchmarking/", "publish_date": "2021-08", "corpus_stage": "推理部署", "notes": "NLP基准测试的挑战与机遇", "id": "4a7a7e8ed437", "collected_at": "2026-10-02T14:15:55", "title": "Challenges and Opportunities in NLP Benchmarking", "body_markdown": "natural language processing\nChallenges and Opportunities in NLP Benchmarking\nSebastian Ruder\n23 Aug 2021\n•\n16 min read\nShare\nOver the last years, models in NLP have become much more powerful, driven by\nadvances in transfer learning\n. A consequence of this drastic increase in performance is that existing benchmarks have been left behind. Recent models \"have outpaced the benchmarks to test for them\" (\nAI Index Report 2021\n), quickly reaching super-human performance on standard benchmarks such as\nSuperGLUE\nand\nSQuAD\n. Does this mean that we have solved natural language processing? Far from it.\nHowever, the traditional practices for evaluating performance of NLP models, using a single metric such as accuracy or BLEU, relying on static benchmarks and abstract task formulations no longer work as well in light of models' surprisingly robust\nsuperficial\nnatural language understanding ability. We thus need to rethink how we design our benchmarks and evaluate our models so that they can still serve as useful indicators of progress going forward.\nThis post aims to give an overview of challenges and opportunities in benchmarking in NLP, together with some general recommendations. I tried to cover perspectives from recent papers, talks at ACL 2021 as well as at the\nACL 2021 Workshop on Benchmarking: Past, Present and Future\n, in addition to some of my own thoughts.\nHeader image: Performance on\nSQuAD 2.0\nover time (Credit:\nPapers with Code\n)\nTable of contents:\nWhat is a benchmark?\nA brief history of benchmarking\nMetrics matter\nConsider the downstream use case\nFine-grained evaluation\nThe long tail of benchmark performance\nLarge-scale continuous evaluation\nWhat is a benchmark?\n\"Datasets are the telescopes of our field.\"\n—\nAravind Joshi\nThe original use of the term refers to horizontal marks made by\nsurveyors\nin stone structures, into which an angle-iron could be placed to form a \"bench\" for a\nleveling rod\n. Figuratively, a benchmark refers to a standard point of reference against which things can be compared. A benchmark as it is used in ML or NLP typically has several components: it consists of one or multiple datasets, one or multiple associated metrics, and a way to aggregate performance.\nA benchmark sets a standard for assessing the performance of different systems that is agreed upon by the community. To ensure that a benchmark is accepted by the community, many recent benchmarks either select a representative set of standard tasks, such as\nGLUE\nor\nXTREME\nor actively solicit task proposals from the community, such as\nSuperGLUE\n,\nGEM\n, or\nBIG-Bench\n.\nFor people in the field, benchmarks are crucial tools to track progress.\nAravind Joshi\nsaid that without benchmarks to assess the performance of our models, we are just like \"astronomers wanting to see the stars but refusing to build telescopes\".\nFor practitioners and outsiders, benchmarks provide an objective lens into a field that enables them to identify useful models and keep track of a field's progress. For instance, the\nAI Index Report 2021\nuses SuperGLUE and SQuAD as a proxy for overall progress in natural language processing.\nReaching human performance on influential benchmarks is often seen as a key milestone for a field. AlphaFold 2 reaching performance competitive with experimental methods on the CASP 14 competition marked a\nmajor scientific advance in the field of structural biology\n.\nA brief history of benchmarking\n\"Creating good benchmarks is harder than most imagine.\"\n—John R. Mashey; foreword to\nSystems Benchmarking (2020)\nBenchmarks have a long history of being used to assess the\nperformance of computational systems\n. The Standard Performance Evaluation Corporation (\nSPEC\n), established in 1988 is one of the oldest organisations dedicated to benchmarking the performance of computer hardware. Crucially, SPEC had support from most important companies in the field. Every year, it would release different benchmark sets, each composed of multiple programs, with performance measured as the\ngeometric mean\nof millions of instructions per second (\nMIPS\n).\nA recent ML-specific analogue to SPEC is\nMLCommons\n, which organises the\nMLPerf\nseries of performance benchmarks focusing on model training and inference. Similar to SPEC, MLPerf has a broad base of support from academia and industry, building on previous individual efforts for measuring performance such as\nDeepBench\nby Baidu or\nDAWNBench\nby Stanford.\nFor US agencies such as\nDARPA\nand\nNIST\n, benchmarks played a crucial role in measuring and tracking scientific progress. Early benchmarks for automatic speech recognition (ASR) such as\nTIMIT\nand\nSwitchboard\nwere funded by DARPA and coordinated by\nNIST\nstarting in 1986. Later influential benchmarks in other areas of ML such as\nMNIST\nwere also based on NIST data.\nFor language technology and information retrieval (IR), NIST ran the DARPA-funded\nTREC\nseries of workshops covering a wide array of tracks and topics, which can be seen below. TREC organised competitions built on an evaluation paradigm pioneered by\nCranfield\nin the 1960s where models are evaluated based on a set of test collections, consisting of documents, questions, and human relevance judgements. As the variance in performance across topics is large, scores are averaged over many topics. TREC's \"standard, widely available, and carefully constructed set of data laid the groundwork for further innovation\" (\nVarian, 2008\n) in IR.\nTasks and topics in the TREC workshops from 1992–2020 (Credit:\nEllen Voorhees\n)\nMany recent influential benchmarks such as\nImageNet\n,\nSQuAD\n, or\nSNLI\nare large in scale, consisting of hundreds of thousands of examples and were developed by academic groups at well-funded universities. In the era of deep learning, such large-scale datasets have been credited as one of the pillars driving progress in research, with fields such as NLP or\nbiology\nwitnessing their\n'ImageNet moment'\n.\nAs models have become more powerful and general-purpose, benchmarks have become more application-oriented and increasingly moved from single-task to multi-task and single-domain to multi-domain benchmarks. Key examples of these trends are a transition from a focus on core linguistic tasks such as\npart-of-speech tagging\nand\ndependency parsing\nto tasks that are closer to the real-world such as goal-oriented dialogue and open-domain question answering (\nKwiatkowski et al., 2019\n); the emergence of multi-task datasets such as\nGLUE\n; and multi-modality datasets such as\nWILDS\n.\nHowever, while it took more than 15 years to achieve superhuman performance on classic benchmarks such as MNIST or Switchboard, models have achieved superhuman performance on more recent benchmarks such as GLUE and SQuAD 2.0 within about a year of their release, as can be seen in the figure below. At the same time, we know that the capabilities that these benchmarks aim to test, such as general question answering are far from being solved.\nBenchmark saturation over time for popular benchmarks. Initial performance and human performance are normalised to -1 and 0 respectively (\nKiela et al., 2021\n).\nAnother factor that has contributed to the saturation of these benchmarks is that limitations and annotation artefacts of recent datasets have been identified much more quickly compared to earlier benchmarks. In SNLI, annotators have been shown to rely on heuristics, which allow models to make the correct prediction in many cases using the hypothesis alone (\nGururangan et al., 2018\n) while models trained on SQuAD are subject to adversarially inserted sentences (\nJia and Liang, 2017\n).\nA recent trend is the development of adversarial datasets such as Adversarial NLI (\nNie et al., 2020\n), Beat the AI (\nBartolo et al., 2020\n), and others where examples are created to be difficult for current models.\nDynabench\n(\nKiela et al., 2021\n), a recent open-source platform has been designed to facilitate the creation of such datasets. A benefit of such benchmarks is that they can be dynamically updated to be challenging as new models emerge and consequently do not saturate as easily as static benchmarks.\nMetrics matter\n\"When you can measure what you are speaking of and express it in numbers, you know that on which you are discussing. But when you cannot measure it and express it in numbers, your knowledge is of a very meagre and unsatisfactory kind.\"\n—Lord Kelvin\nWhen it comes to measuring performance, metrics play an important and often under-appreciated role. For classification tasks, accuracy or\nF-score\nmetrics may seem like the obvious choice but—depending on the application—different types of errors incur different costs. For fine-grained sentiment analysis, confusing between\npositive\nand\nvery positive\nmay not be problematic while mixing up\nvery positive\nand\nvery negative\nis. Chris Potts highlights an\narray of practical examples\nwhere metrics like F-score fall short, many in scenarios where errors are much more costly.\nDesigning a good metric requires domain expertise.\nMLPerf\nmeasures the wallclock time required to train a model to a dataset-specific performance target, a measure informed by both end use cases and the difficulty to compare other efficiency metrics such as FLOPS across models. In ASR, only the percentage of correctly transcribed words (akin to accuracy) was initially used as the metric. The community later settled on\nword error rate\n, i.e. $\\frac{\\text{substitutions} + \\text{deletions} + \\text{insertions}}{\\text{number of words in reference}}$ as it directly reflects the cost of correcting transcription errors.\nThere is a large difference between metrics designed for decades-long research and metrics designed for near-term development of practical applications, as highlighted by\nMark Liberman\n. For developing decade-scale technology, we need efficient metrics that can be crude as long as they point in the general direction of our distant goal. Examples of such metrics are the word error rate in ASR (which assumes that all words are equally important) and BLEU in machine translation (which assumes that word order is not important). In contrast, for the evaluation of practical technology we need metrics that are designed with the requirements of specific applications in mind and that can consider different types of error classes.\nThe rapid increase in model performance in recent years has catapulted us from the decade-long to the near-term regime for many applications. However, even in this more application-oriented setting we are still relying on the same metrics that we have used to measure long-term research progress thus far. In a recent meta-analysis,\nMarie et al. (2021)\nfound that 82% papers of machine translation (MT) papers between 2019–2020 only evaluate on BLEU—despite 108 alternative metrics having been proposed for MT evaluation in the last decade, many of which correlate better with human judgements. As models become stronger, metrics like BLEU are no longer able to accurately identify and compare the best-performing models.\nWhile evaluation of natural language generation (NLG) models is notoriously difficult, standard n-gram overlap-based metrics such as ROUGE or BLEU are furthermore less suited to languages with rich morphology, which will be assigned relatively lower scores.\nA recent trend in NLG is towards the development of automatic metrics such as BERTScore (\nZhang et al., 2020\n) that leverage the power of large pre-trained models. A recent modification of this method makes it more suitable for near-term MT evaluation by assigning larger weights to more difficult tokens, i.e. tokens that are translated correctly only by few MT systems (\nZhan et al., 2021\n).\nIn order to continue to make progress, we need to be able to update and refine our metrics, to replace efficient simplified metrics with application-specific ones. The recent\nGEM benchmark\n, for instance, explicitly includes\nmetrics\nas a component that should be improved over time, as can be seen below.\nOpportunities (circle) and challenges of benchmark evaluation (\nGehrmann et al., 2021\n).\nRecommendations:\nConsider metrics that are better suited to the downstream task and language.\nConsider metrics that highlight the trade-offs of the downstream setting.\nUpdate and refine metrics over time.\nConsider the downstream use case\n\"[...] benchmarks shape a field, for better or worse. Good benchmarks are in alignment with real applications, but bad benchmarks are not, forcing engineers to choose between making changes that help end users or making changes that only help with marketing.\"\n—David A. Patterson; foreword to\nSystems Benchmarking (2020)\nNLP technology is increasingly used in many real-world application areas,\nfrom creative self-expression to fraud detection and recommendation\n. It is thus key to design benchmarks with such real-world settings in mind.\nA benchmark's data composition and evaluation protocol should reflect the real-world use case, as highlighted by\nIdo Dagan\n. For relation classification, for instance, the\nFewRel\ndataset\nlacks some important realistic properties\n, which\nFew-shot TACRED\naddresses. For binary sentiment classification on the\nIMDb dataset\n, only highly polarised positive and negative reviews are considered and labels are exactly balanced. In information retrieval, retrieving relevant before non-relevant documents is\nnecessary\nbut not sufficient\nfor real-world usage\n.\nAs a first rule of social responsibility for NLP,\nChris Potts proposes\n\"Do exactly what you said you would do\". As researchers in the field, we should communicate clearly what performance on a benchmark reflects and how this corresponds to real-world settings. In a similar vein,\nBowman and Dahl (2021)\nargue that good performance on a benchmark should imply robust in-domain performance on the task.\nHowever, the real-world application of a task may confront the model with data different from its training distribution. It is thus key to assess the robustness of the model and how well it generalises to such out-of-distribution data, including data with a temporal shift and data from other language varieties.\nIn light of the limited linguistic diversity in NLP research (\nJoshi et al., 2020\n), it is furthermore crucial not to treat English as the singular language for evaluation. When designing a benchmark, collecting—at a minimum—test data in other languages may help to highlight new challenges and promote language inclusion. Similarly, when evaluating models, leveraging the increasing number of non-English language datasets in tasks such as\nquestion answering\nand summarisation (\nHasan et al., 2021\n) can provide additional evidence of a model's versatility.\nUltimately, considering the challenges of current and future real-world applications of language technology may provide inspiration for many new evaluations and benchmarks. Benchmarks are among the most impactful artefacts of our field and often lead to entirely new research directions, so it is crucial for them to reflect real-world and potentially ambitious use cases of our technology.\nRecommendations:\nDesign the benchmark and its evaluation so that it reflects the real-world use case.\nEvaluate in-domain and out-of-domain generalisation.\nCollect data and evaluate models on other languages.\nTake inspiration from real-world applications of language technology.\nFine-grained evaluation\n\"No matter how much people want performance to be a single number, even the\nright\nmean with no distribution can be misleading, and the\nwrong\nmean certainly is no better.\"\n—John R. Mashey\nThe downstream use case of technology should also inform the metrics we use for evaluation. In particular, for downstream applications often not a single metric but an array of constraints need to be considered. Rada Mihalcea\ncalls for moving away from just focusing on accuracy\nand to focus on other important aspects of real-world scenarios. What is important in a particular setting, in other words, the utility of an NLP system, ultimately depends on the requirements of each individual user (\nEthayarajh and Jurafsky, 2020\n).\nSocietal needs have generally not been emphasised in machine learning research (\nBirhane et al., 2021\n). However, for real-world applications it is particularly crucial that a model does not exhibit any harmful social biases. Testing for such biases in a task-specific manner should thus become a standard part of algorithm development and model evaluation.\nAnother aspect that is important for practical applications is efficiency. Depending on the application, this can relate to both sample efficiency,\nFLOPS\n, and memory constraints. Evaluating models in resource-constrained settings can often lead to new research directions. For instance, the EfficientQA competition (\nMin et al., 2020\n) at NeurIPS 2020 demonstrated the benefits of\nretrieval augmentation\nand large collections of weakly supervised question–answer pairs (\nLewis et al., 2021\n).\nIn order to better understand the strengths and weaknesses of our models, we furthermore require more fine-grained evaluation across a\nsingle\nmetric, highlighting on what types of examples models excel and fail at.\nExplainaBoard\n(\nLiu et al., 2021\n) implements such a fine-grained breakdown of model performance across different tasks, which can be seen below. Another way to obtain a more fine-grained estimate of model performance is to create test cases for specific phenomena and model behaviour, for instance using the CheckList framework (\nRibeiro et al., 2020\n).\nThe\nExplainaBoard interface for the CoNLL-2003 NER dataset\nfor the three best systems including single-system analyses for the best system (A), pairwise analysis results for the top-2 systems (B), a common error table (C), and combined results (D) (\nLiu et al., 2021\n).\nAs individual metrics can be flawed, it is key to evaluate across multiple metrics. When evaluating on multiple metrics, scores are typically averaged to obtain a single score. A single score is useful to compare models at a glance and provides people outside the community a clear way to assess model performance. However, using the arithmetic mean is not appropriate for all purposes. SPEC used the geometric mean, $\\sqrt[\\leftroot{-2}\\uproot{2}n]{x_1 x_2 \\ldots x_n}$, which is useful when aggregating values that are exponential in nature, such as runtimes.\nAn alternative is to use a weighted sum and to enable the user to define their own weights for each component.\nDynaBench\nuses such a dynamic weighting to weight the importance of model performance but also consider model throughput, memory consumption, fairness, and robustness, which enables users to effectively define their own leaderboard (\nEthayarajh and Jurafsky, 2020\n), as can be seen below.\nDynamic metric weighting in the\nDynaBench natural language inference task leaderboard\nRecommendations:\nMove away from using a single metric for performance evaluation.\nEvaluate social bias and efficiency.\nPerform a fine-grained evaluation of models.\nConsider how to aggregate multiple metrics.\nThe long tail of benchmark performance\nGiven that current models perform surprisingly well on in-distribution examples, it is time to shift our attention to the tail of the distribution, to outliers and atypical examples. Rather than considering only the average case, we should care more about the worst case and subsets of our data where our models perform the worst.\nAs models become more powerful, the fraction of examples where the performance of models differs and that thus will be able to differentiate between strong and the best models will grow smaller. To ensure that evaluation on this long tail of examples is reliable, benchmarks need to be large enough so that small differences in performance can be detected. It is important to note that larger models are not uniformly better across all examples (\nZhong et al., 2021\n).\nAs an alternative, we can develop mechanisms that allow us to identify the best systems with few examples. This is particularly crucial in settings where assessing performance of many systems is expensive, such as in human evaluation for natural language generation.\nMendonça et al. (2021)\nframe this as an online learning problem in the context of MT.\nBenchmarks can also focus on annotating examples that are much more challenging. This is the direction taken by recent adversarial benchmarks (\nKiela et al., 2021\n). Such benchmarks, as long as they are not biased towards a specific model, can be a useful complement to regular benchmarks that sample from the natural distribution. These directions benefit from the development of\nactive\nevaluation methods to identify or generate the most salient and discriminative examples to assess model performance as well as interpretability methods to allow annotators to better understand models' decision boundaries.\nAs the budget (and thus size) of benchmarks generally remains constant, statistical significance testing becomes even more important as it enables us to reliably detect qualitatively relevant performance differences between systems.\nIn order to perform reliable comparisons, the benchmark's annotations should be correct and reliable. However, as models become more powerful, many instances of what look like model errors may be genuine examples of ambiguity in the data.\nBowman and Dahl (2021)\nhighlight how a model may exploit clues about such disagreements to reach super-human performance on a benchmark.\nIf possible, benchmarks should aim to collect multiple annotations to identify ambiguous examples. Such information may provide a useful learning signal (\nPlank et al., 2014\n) and can be helpful for error analysis. In light of such ambiguity, it is even more important to report standard metrics such as inter-annotator agreement as this provides a ceiling for a model's performance on a benchmark.\nRecommendations:\nInclude many and/or hard examples in the benchmark.\nConduct statistical significance testing.\nCollect multiple annotations for ambiguous examples.\nReport inter-annotator agreement.\nLarge-scale continuous evaluation\n\"When a measure becomes a target, it ceases to be a good measure.\"\n—Goodhart's law\nMulti-task benchmarks such as GLUE have become key indicators of progress but such static benchmark collections quickly become outdated. Modelling advances generally also do not lead to uniform progress across tasks. While models have achieved super-human performance on most GLUE tasks, a gap to 5-way human agreement remains on some tasks such as CoLA (\nNangia and Bowman, 2019\n). On\nXTREME\n, models have improved much more on cross-lingual retrieval.\nIn light of the fast pace of model improvements, we are in need of more nimble mechanisms for model evaluation. Specifically, beyond dynamic\nsingle-task\nevaluations such as\nDynaBench\n, it would be useful to define a dynamic\ncollection\nof benchmark datasets on which models have not reached human performance. This collection should be managed by the community, with datasets removed or down-weighted as models reach human performance and new challenging datasets being regularly added. Such a collection needs to be versioned, to enable updates beyond the cycle of academic review and to enable replicability and comparison to prior approaches.\nExisting multi-task benchmarks such as\nGEM\n(\nGehrmann et al., 2021\n), which explicitly aims to be a 'living' benchmark, generally include around 10–15 different tasks. Rather than limiting the benchmark to a small collection of representative tasks, in light of the number of new datasets constantly being released, it might be more useful to include a larger cross-section of NLP tasks. Given the diverse nature of tasks in NLP, this would provide a more robust and up-to-date evaluation of model performance.\nLUGE\nby Baidu is a step towards such a large collection of tasks for Chinese natural language processing, currently consisting of 28 datasets.\nThe collection of tasks can be broken down in various ways, providing more a fine-grained assessment of model capabilities. Such a breakdown may be particularly insightful if tasks or subsets of task data are categorised according to the behaviour they are testing.\nBIG-Bench\n, a recent collaborative benchmark for language model probing includes a\ncategorisation by keyword\n.\nA key challenge for such large-scale multi-task evaluation is accessibility. Tasks need to be available in a common input format so that they can be run easily. In addition, tasks should be efficient to run or alternatively infrastructure needs to be available to run tasks even without much compute.\nAnother point of consideration is that such a collection favours large general-purpose models, which are generally trained by deep-pocketed companies or institutions. Such models, however, are already used as the starting point for most current research efforts and can be—once trained—more efficiently used via\nfine-tuning\n, distillation, or pruning.\nRecommendations:\nConsider collecting and evaluating on a large, diverse, versioned collection of NLP tasks.\nConclusion\nIn order to keep up with advances in modelling, we need to revisit many tacitly accepted benchmarking practices such as relying on simplistic metrics like F1-score and BLEU. To this end, we should take inspiration from real-world applications of language technology and consider the constraints and requirements that such settings pose for our models. We should also care more about the long tail of the distribution as that is where improvements will be observed for many applications. Lastly, we should be more rigorous in the evaluation on our models and rely on multiple metrics and statistical significance testing, contrary to\ncurrent trends\n.\nCitation\nFor attribution in academic contexts, please cite this work as:\n@misc{ruder2021benchmarking,\n author = {Ruder, Sebastian},\n title = {{Challenges and Opportunities in NLP Benchmarking}},\n year = {2021},\n howpublished = {\\url{http://ruder.io/nlp-benchmarking}},\n}", "metadata": { "title": "Challenges and Opportunities in NLP Benchmarking", "description": "Recent NLP models have outpaced the benchmarks to test for them. This post provides an overview of challenges and opportunities for NLP benchmarks.", "keywords": null, "author": null, "og:site_name": "ruder.io", "og:type": "article", "og:title": "Challenges and Opportunities in NLP Benchmarking", "og:description": "Recent NLP models have outpaced the benchmarks to test for them. This post provides an overview of challenges and opportunities for NLP benchmarks.", "og:url": "https://www.ruder.io/nlp-benchmarking/", "og:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/08/squad_2_progress.png", "og:image:width": "1162", "og:image:height": "499", "twitter:card": "summary_large_image", "twitter:title": "Challenges and Opportunities in NLP Benchmarking", "twitter:description": "Recent NLP models have outpaced the benchmarks to test for them. This post provides an overview of challenges and opportunities for NLP benchmarks.", "twitter:url": "https://www.ruder.io/nlp-benchmarking/", "twitter:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/08/squad_2_progress.png", "twitter:label1": "Written by", "twitter:data1": "Sebastian Ruder", "twitter:label2": "Filed under", "twitter:data2": "natural language processing, transfer learning", "twitter:site": "@seb_ruder", "article:published_time": "2021-08-23T09:00:00.000Z", "article:modified_time": "2021-08-23T15:45:06.000Z", "article:tag": "transfer learning" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/4a7a7e8ed437/code/body-000.txt", "chars": 201, "language": "text" } ], "images": [ { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2021/08/squad_2_progress.png", "bytes": 37447, "reused": true, "local_path": "attachments/4a7a7e8ed437/images/001.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2021/08/squad_2_progress.png", "alt": "Challenges and Opportunities in NLP Benchmarking", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/08/trec_series.png", "bytes": 481426, "reused": true, "local_path": "attachments/4a7a7e8ed437/images/005.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/08/trec_series.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/08/dynabench_plot.png", "bytes": 287560, "reused": false, "local_path": "attachments/4a7a7e8ed437/images/body-6aab6fc543e819efbaa1.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/08/dynabench_plot.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/08/opportunities_of_benchmarks.png", "bytes": 95709, "reused": false, "local_path": "attachments/4a7a7e8ed437/images/body-53ff7da2c6b0734a4b39.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/08/opportunities_of_benchmarks.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/08/explainaboard_leaderboard-1.png", "bytes": 356892, "reused": false, "local_path": "attachments/4a7a7e8ed437/images/body-d15e6405dea47d5aa041.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/08/explainaboard_leaderboard-1.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/08/dynabench_weighting.png", "bytes": 138207, "reused": false, "local_path": "attachments/4a7a7e8ed437/images/body-4169a3345347997b877b.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/08/dynabench_weighting.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/watch?v=t_A36DDcG_0", "src": "https://youtu.be/t_A36DDcG_0?t=964&ref=ruder.io", "status": "ok", "kind": "single_video", "local_path": "attachments/4a7a7e8ed437/videos/repair-t_A36DDcG_0.mp4", "bytes": 83073327, "valid": true, "height": 1080, "width": 1920, "duration": "2755.941000", "reused": true, "reused_from": "xinxing-zimeiti-corpus/attachments/1950092ff420/videos/t_A36DDcG_0.mp4" }, { "url": "https://www.youtube.com/watch?v=a-ukPup8gKw", "src": "https://youtu.be/a-ukPup8gKw?t=770&ref=ruder.io", "status": "ok", "kind": "single_video", "local_path": "attachments/4a7a7e8ed437/videos/repair-a-ukPup8gKw.mp4", "bytes": 38750622, "valid": true, "height": 1080, "width": 1728, "duration": "921.920000", "reused": false }, { "url": "https://www.youtube.com/watch?v=Jb6G25ZJ-VU", "src": "https://youtu.be/Jb6G25ZJ-VU?t=307&ref=ruder.io", "status": "no_1080_or_720", "kind": "single_video", "error": "ERROR: [youtube] Jb6G25ZJ-VU: Requested format is not available. Use --list-formats for a list of available formats\n", "title": "", "retried_at": "2026-10-20T17:41:00Z" }, { "url": "https://www.youtube.com/watch?v=FXCSWvIsdEE", "src": "https://youtu.be/FXCSWvIsdEE?t=1226&ref=ruder.io", "status": "ok", "kind": "single_video", "local_path": "attachments/4a7a7e8ed437/videos/repair-FXCSWvIsdEE.mp4", "bytes": 46269711, "valid": true, "height": 720, "width": 1280, "duration": "1284.041000", "reused": false } ], "links": [ { "href": "https://www.ruder.io/tag/natural-language-processing/", "label": "natural language processing", "type": "a" }, { "href": "https://www.ruder.io/author/sebastian/", "label": "", "type": "a" }, { "href": "https://docs.google.com/presentation/d/1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc/edit?usp=sharing&ref=ruder.io", "label": "advances in transfer learning", "type": "a" }, { "href": "https://aiindex.stanford.edu/wp-content/uploads/2021/03/2021-AI-Index-Report_Master.pdf?ref=ruder.io", "label": "AI Index Report 2021", "type": "a" }, { "href": "https://super.gluebenchmark.com/?ref=ruder.io", "label": "SuperGLUE", "type": "a" }, { "href": "https://rajpurkar.github.io/SQuAD-explorer/?ref=ruder.io", "label": "SQuAD", "type": "a" }, { "href": "https://github.com/kwchurch/Benchmarking_past_present_future?ref=ruder.io", "label": "ACL 2021 Workshop on Benchmarking: Past, Present and Future", "type": "a" }, { "href": "https://paperswithcode.com/sota/question-answering-on-squad20?ref=ruder.io", "label": "Papers with Code", "type": "a" }, { "href": "https://youtu.be/t_A36DDcG_0?t=964&ref=ruder.io", "label": "Aravind Joshi", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Surveying?ref=ruder.io", "label": "surveyors", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Level_staff?ref=ruder.io", "label": "leveling rod", "type": "a" }, { "href": "https://gluebenchmark.com/?ref=ruder.io", "label": "GLUE", "type": "a" }, { "href": "https://github.com/google-research/xtreme?ref=ruder.io", "label": "XTREME", "type": "a" }, { "href": "https://gem-benchmark.com/?ref=ruder.io", "label": "GEM", "type": "a" }, { "href": "https://github.com/google/BIG-bench?ref=ruder.io", "label": "BIG-Bench", "type": "a" }, { "href": "https://www.youtube.com/watch?t=964&v=t_A36DDcG_0&feature=youtu.be&ref=ruder.io", "label": "Aravind Joshi", "type": "a" }, { "href": "https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology?ref=ruder.io", "label": "major scientific advance in the field of structural biology", "type": "a" }, { "href": "https://www.springer.com/gp/book/9783030417048?ref=ruder.io", "label": "Systems Benchmarking (2020)", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Benchmark_(computing)?ref=ruder.io", "label": "performance of computational systems", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Standard_Performance_Evaluation_Corporation?ref=ruder.io", "label": "SPEC", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Geometric_mean?ref=ruder.io", "label": "geometric mean", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Instructions_per_second?ref=ruder.io", "label": "MIPS", "type": "a" }, { "href": "https://mlcommons.org/en/?ref=ruder.io", "label": "MLCommons", "type": "a" }, { "href": "https://mlcommons.org/en/training-normal-10/?ref=ruder.io", "label": "MLPerf", "type": "a" }, { "href": "https://github.com/baidu-research/DeepBench?ref=ruder.io", "label": "DeepBench", "type": "a" }, { "href": "https://dawn.cs.stanford.edu/benchmark/?ref=ruder.io", "label": "DAWNBench", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/DARPA?ref=ruder.io", "label": "DARPA", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/National_Institute_of_Standards_and_Technology?ref=ruder.io", "label": "NIST", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/TIMIT?ref=ruder.io", "label": "TIMIT", "type": "a" }, { "href": "https://catalog.ldc.upenn.edu/LDC97S62?ref=ruder.io", "label": "Switchboard", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/MNIST_database?ref=ruder.io", "label": "MNIST", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Text_Retrieval_Conference?ref=ruder.io", "label": "TREC", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Cranfield_experiments?ref=ruder.io", "label": "Cranfield", "type": "a" }, { "href": "https://googleblog.blogspot.com/2008/03/why-data-matters.html?ref=ruder.io", "label": "Varian, 2008", "type": "a" }, { "href": "https://www.nist.gov/video/coopetition-ir-research-presented-ellen-voorhees?ref=ruder.io", "label": "Ellen Voorhees", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/ImageNet?ref=ruder.io", "label": "ImageNet", "type": "a" }, { "href": "https://aclanthology.org/D15-1075/?ref=ruder.io", "label": "SNLI", "type": "a" }, { "href": "https://twitter.com/OriolVinyalsML/status/1333436710303264772?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1333436710303264772%7Ctwgr%5E%7Ctwcon%5Es1_c10&ref_url=https%3A%2F%2Fsyncedreview.com%2F2020%2F11%2F30%2Fbiologys-imagenet-moment-deepmind-says-its-alphafold-has-cracked-a-50-year-old-biology-challenge%2F&ref=ruder.io", "label": "biology", "type": "a" }, { "href": "https://thegradient.pub/nlp-imagenet/?ref=ruder.io", "label": "'ImageNet moment'", "type": "a" }, { "href": "http://nlpprogress.com/english/part-of-speech_tagging.html?ref=ruder.io", "label": "part-of-speech tagging", "type": "a" }, { "href": "http://nlpprogress.com/english/dependency_parsing.html?ref=ruder.io", "label": "dependency parsing", "type": "a" }, { "href": "https://research.google/pubs/pub47761/?ref=ruder.io", "label": "Kwiatkowski et al., 2019", "type": "a" }, { "href": "https://wilds.stanford.edu/?ref=ruder.io", "label": "WILDS", "type": "a" }, { "href": "https://aclanthology.org/2021.naacl-main.324.pdf?ref=ruder.io", "label": "Kiela et al., 2021", "type": "a" }, { "href": "https://aclanthology.org/N18-2017.pdf?ref=ruder.io", "label": "Gururangan et al., 2018", "type": "a" }, { "href": "https://aclanthology.org/D17-1215/?ref=ruder.io", "label": "Jia and Liang, 2017", "type": "a" }, { "href": "https://aclanthology.org/2020.acl-main.441/?ref=ruder.io", "label": "Nie et al., 2020", "type": "a" }, { "href": "https://aclanthology.org/2020.tacl-1.43.pdf?ref=ruder.io", "label": "Bartolo et al., 2020", "type": "a" }, { "href": "https://dynabench.org/?ref=ruder.io", "label": "Dynabench", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/F-score?ref=ruder.io", "label": "F-score", "type": "a" }, { "href": "https://youtu.be/t_A36DDcG_0?t=1824&ref=ruder.io", "label": "array of practical examples", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Word_error_rate?ref=ruder.io", "label": "word error rate", "type": "a" }, { "href": "https://youtu.be/a-ukPup8gKw?t=770&ref=ruder.io", "label": "Mark Liberman", "type": "a" }, { "href": "https://aclanthology.org/2021.acl-long.566.pdf?ref=ruder.io", "label": "Marie et al. (2021)", "type": "a" }, { "href": "https://openreview.net/forum?id=SkeHuCVFDr&ref=ruder.io", "label": "Zhang et al., 2020", "type": "a" }, { "href": "https://aclanthology.org/2021.acl-short.5.pdf?ref=ruder.io", "label": "Zhan et al., 2021", "type": "a" }, { "href": "https://github.com/GEM-benchmark/GEM-metrics?ref=ruder.io", "label": "metrics", "type": "a" }, { "href": "https://aclanthology.org/2021.gem-1.10.pdf?ref=ruder.io", "label": "Gehrmann et al., 2021", "type": "a" }, { "href": "https://youtu.be/t_A36DDcG_0?t=278&ref=ruder.io", "label": "from creative self-expression to fraud detection and recommendation", "type": "a" }, { "href": "https://youtu.be/Jb6G25ZJ-VU?t=307&ref=ruder.io", "label": "Ido Dagan", "type": "a" }, { "href": "https://aclanthology.org/D18-1514/?ref=ruder.io", "label": "FewRel", "type": "a" }, { "href": "https://youtu.be/Jb6G25ZJ-VU?t=657&ref=ruder.io", "label": "lacks some important realistic properties", "type": "a" }, { "href": "https://arxiv.org/abs/2104.08481?ref=ruder.io", "label": "Few-shot TACRED", "type": "a" }, { "href": "https://ai.stanford.edu/~ang/papers/acl11-WordVectorsSentimentAnalysis.pdf?ref=ruder.io", "label": "IMDb dataset", "type": "a" }, { "href": "https://youtu.be/t_A36DDcG_0?t=661&ref=ruder.io", "label": "Chris Potts proposes", "type": "a" }, { "href": "https://aclanthology.org/2021.naacl-main.385.pdf?ref=ruder.io", "label": "Bowman and Dahl (2021)", "type": "a" }, { "href": "https://aclanthology.org/2020.acl-main.560/?ref=ruder.io", "label": "Joshi et al., 2020", "type": "a" }, { "href": "https://nlpprogress.com/?ref=ruder.io", "label": "question answering", "type": "a" }, { "href": "https://aclanthology.org/2021.findings-acl.413/?ref=ruder.io", "label": "Hasan et al., 2021", "type": "a" }, { "href": "https://youtu.be/FXCSWvIsdEE?t=1226&ref=ruder.io", "label": "calls for moving away from just focusing on accuracy", "type": "a" }, { "href": "https://aclanthology.org/2020.emnlp-main.393/?ref=ruder.io", "label": "Ethayarajh and Jurafsky, 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2106.15590?ref=ruder.io", "label": "Birhane et al., 2021", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/FLOPS?ref=ruder.io", "label": "FLOPS", "type": "a" }, { "href": "https://arxiv.org/abs/2101.00133?ref=ruder.io", "label": "Min et al., 2020", "type": "a" }, { "href": "https://ruder.io/research-highlights-2020/?ref=ruder.io", "label": "retrieval augmentation", "type": "a" }, { "href": "https://arxiv.org/abs/2102.07033?ref=ruder.io", "label": "Lewis et al., 2021", "type": "a" }, { "href": "http://explainaboard.nlpedia.ai/?ref=ruder.io", "label": "ExplainaBoard", "type": "a" }, { "href": "https://aclanthology.org/2021.acl-demo.34/?ref=ruder.io", "label": "Liu et al., 2021", "type": "a" }, { "href": "https://aclanthology.org/2020.acl-main.442/?ref=ruder.io", "label": "Ribeiro et al., 2020", "type": "a" }, { "href": "http://explainaboard.nlpedia.ai/leaderboard/task-ner/index.php?ref=ruder.io", "label": "ExplainaBoard interface for the CoNLL-2003 NER dataset", "type": "a" }, { "href": "https://aclanthology.org/2021.acl-demo.34.pdf?ref=ruder.io", "label": "Liu et al., 2021", "type": "a" }, { "href": "https://dynabench.org/tasks/nli?ref=ruder.io", "label": "DynaBench natural language inference task leaderboard", "type": "a" }, { "href": "https://aclanthology.org/2021.findings-acl.334.pdf?ref=ruder.io", "label": "Zhong et al., 2021", "type": "a" }, { "href": "https://aclanthology.org/2021.acl-long.242/?ref=ruder.io", "label": "Mendonça et al. (2021)", "type": "a" }, { "href": "https://aclanthology.org/P14-2083.pdf?ref=ruder.io", "label": "Plank et al., 2014", "type": "a" }, { "href": "https://aclanthology.org/P19-1449.pdf?ref=ruder.io", "label": "Nangia and Bowman, 2019", "type": "a" }, { "href": "https://sites.research.google/xtreme/?ref=ruder.io", "label": "XTREME", "type": "a" }, { "href": "https://arxiv.org/abs/2102.01672?ref=ruder.io", "label": "Gehrmann et al., 2021", "type": "a" }, { "href": "https://www.luge.ai/?ref=ruder.io", "label": "LUGE", "type": "a" }, { "href": "https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/keywords_to_tasks.md?ref=ruder.io", "label": "categorisation by keyword", "type": "a" }, { "href": "https://ruder.io/recent-advances-lm-fine-tuning/?ref=ruder.io", "label": "fine-tuning", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:51:59Z", "source_url": "https://www.ruder.io/nlp-benchmarking/", "papers": [ { "status": "http_403", "url": "https://aiindex.stanford.edu/wp-content/uploads/2021/03/2021-AI-Index-Report_Master.pdf?ref=ruder.io", "retried_at": "2026-10-20T17:40:55Z" }, { "status": "ok", "url": "https://aclanthology.org/2021.naacl-main.324.pdf?ref=ruder.io", "bytes": 480678, "reused": true, "local_path": "papers/repair/43864c278284d54a6b91.pdf" }, { "status": "ok", "url": "https://aclanthology.org/N18-2017.pdf?ref=ruder.io", "bytes": 252271, "reused": false, "local_path": "papers/repair/a1bcf8920c7e079b1638.pdf" }, { "status": "ok", "url": "https://aclanthology.org/2020.tacl-1.43.pdf?ref=ruder.io", "bytes": 1217787, "reused": false, "local_path": "papers/repair/c3dcb36e4585f9ce6fdf.pdf" }, { "status": "ok", "url": "https://aclanthology.org/2021.acl-long.566.pdf?ref=ruder.io", "bytes": 346356, "reused": false, "local_path": "papers/repair/40a5ad9ec9580d814c6d.pdf" }, { "status": "ok", "url": "https://aclanthology.org/2021.acl-short.5.pdf?ref=ruder.io", "bytes": 466345, "reused": false, "local_path": "papers/repair/e2d78380df1046ab9b4c.pdf" }, { "status": "ok", "url": "https://aclanthology.org/2021.gem-1.10.pdf?ref=ruder.io", "bytes": 1335845, "reused": false, "local_path": "papers/repair/fd28c41c71b2c2cff81f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2104.08481", "bytes": 566587, "reused": false, "local_path": "papers/repair/48b2679f62ffba725209.pdf" }, { "status": "ok", "url": "https://ai.stanford.edu/~ang/papers/acl11-WordVectorsSentimentAnalysis.pdf?ref=ruder.io", "bytes": 197893, "reused": false, "local_path": "papers/repair/59d24a62af66c37b10ce.pdf" }, { "status": "ok", "url": "https://aclanthology.org/2021.naacl-main.385.pdf?ref=ruder.io", "bytes": 264799, "reused": false, "local_path": "papers/repair/88946ec87ee8c95a249e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2106.15590", "bytes": 1268418, "reused": false, "local_path": "papers/repair/e8d2abffbc16cf76e058.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2101.00133", "bytes": 2904096, "reused": false, "local_path": "papers/repair/a3139cdfd88862860aa5.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2102.07033", "bytes": 756846, "reused": false, "local_path": "papers/repair/86ec95dfde68dafe6808.pdf" }, { "status": "ok", "url": "https://aclanthology.org/2021.acl-demo.34.pdf?ref=ruder.io", "bytes": 1832923, "reused": false, "local_path": "papers/repair/bea7ce15d76ee44b8d7d.pdf" }, { "status": "ok", "url": "https://aclanthology.org/2021.findings-acl.334.pdf?ref=ruder.io", "bytes": 1365826, "reused": false, "local_path": "papers/repair/fa69b1e5d585ac16839a.pdf" }, { "status": "ok", "url": "https://aclanthology.org/P14-2083.pdf?ref=ruder.io", "bytes": 249478, "reused": false, "local_path": "papers/repair/4e53571853cab18b7ca6.pdf" }, { "status": "ok", "url": "https://aclanthology.org/P19-1449.pdf?ref=ruder.io", "bytes": 559243, "reused": false, "local_path": "papers/repair/1fe902873e21bb90fdb2.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2102.01672", "bytes": 1320940, "reused": false, "local_path": "papers/repair/1547ee8ccd0bebd610ff.pdf" } ], "actual_video_files": [], "body_chars": 25971, "expected_body_images": 6, "discovery_status": "ok", "delivery_status": "partial", "missing_reason": "http_403；no_1080_or_720", "raw_html": "attachments/4a7a7e8ed437/pages/source.html", "local_html": "attachments/4a7a7e8ed437/pages/article.html", "body_base_path": "attachments/4a7a7e8ed437/pages/", "body_markdown_path": "attachments/4a7a7e8ed437/pages/article.md" }