{ "source_file": "0930", "source_platform": "个人博客", "author": "Sebastian Ruder", "title_listed": "EMNLP 2023 Primer", "keywords": "EMNLP,NLP,论文", "domain": "文本处理", "content_type": "行业分析", "is_paid_flag": "否", "url": "https://www.ruder.io/emnlp-2023-primer/", "publish_date": "2023-12", "corpus_stage": "预训练/SFT", "notes": "EMNLP 2023论文概览", "id": "d3fbf4449afb", "collected_at": "2026-10-02T14:16:31", "title": "EMNLP 2023 Primer", "body_markdown": "EMNLP 2023 Primer\nAn overview of EMNLP 2023 papers covering QA, instruction tuning, task adaptation, NLG evaluation, and multilingual models and datasets.\nSebastian Ruder\n05 Dec 2023\n•\n11 min read\nShare\nThe tasks in XTREME-UP (\nRuder et al., 2023\n) and their role in language technology.\nEMNLP 2023\n, one of the biggest NLP conferences takes place this week from Dec 6–10 in Singapore.\nIn this post, I’ll discuss a selection of exciting papers and workshops I’m looking forward to at the conference. Here are the main trends I observed (based on the small sample of papers I discuss here and those I came across online):\nInstruction-tuned LMs and LLMs are everywhere.\nSimilar to\nearlier years\nwhere BERT was ubiquitous, instruction-tuned language models (LMs) and large language models (LLMs) are used in almost every paper.\nEvaluation based on LLMs is increasingly common.\nWhile some papers employ automatic evaluation based on GPT-4, new metrics that are proposed are based on LLMs in zero-shot prompted or fine-tuned settings.\nPrompt usage is getting more creative.\nBeyond a standard prompt template, prompts are getting increasingly complex and specialized to the desired setting. Techniques such as\nchain-of-thought prompting\nare common tools.\nMultilinguality is increasingly popular.\nI came across a substantial number of papers studying multilingual settings, which indicates that LLMs are still limited in non-English settings and that making LLMs more multilingual is an important direction.\nOn the other hand, I did not come across many papers that tried to analyze LLM properties (using a synthetic setup, for instance) or that used external models or tools to augment LLMs (\nplease point me to papers that I missed\n). This is in contrast to NeurIPS 2023 where\nsuch papers were more common\n.\nPapers with a † are presented in Findings of EMNLP (rather than the main conference). I am an author/co-organizer on papers/events indicated with a *.\nThis post was first published in\nNLP News\n.\nWorkshops\nGenBench Workshop\n, Dec 6.\nGeneralization is crucial to ensure robust model behavior but how good generalization looks like and how it should be evaluated is still not well understood. The GenBench workshop on (benchmarking) generalization in NLP aims to catalyse research on generalization and how to measure it in the NLP community. Accepted papers study generalization or are\nBIG-bench\n-style\ncollaborative benchmarking tasks\n(CBT). The\nprogram\nconsists of invited talks, CBT spotlights as well as oral presentations and posters.\nWorkshop for NLP Open-Source Software (NLP-OSS)\n, Dec 6.\nIn light of the increasing number of closed-source LLMs, it is important to continue to promote an open culture of sharing knowledge, data, and software, from which the NLP community has benefited greatly. This workshop aims to further the sharing of insights regarding the creation and development of NLP open-source software. Invited talks feature important NLP open-source projects including\ntrlX\n, a framework for large-scale open-source RLHF and\nSEA-LION\n, LLMs pre-trained for Southeast Asian languages.\nThe Big Picture Workshop: Crafting a Research Narrative\n, Dec 7\n*.  In research, we “stand on the shoulders of giants”. However, given the number and rapid pace of published papers, it has become increasingly difficult, to recognize the larger story to which a paper is connected. The Big Picture Workshop aims to explore and distill such broader research narratives. We have a\ndiverse set of accepted papers\nthat provide insightful syntheses of different threads of research. On the workshop day, we’ll try out a new presentation format where we have researchers from different groups working on the same topic critically reflect on and discuss their work.\nMultilingual Representation Learning Workshop (MRL)\n, Dec 7*\n. This workshop provides a forum to discuss work to improve NLP in low-resource and under-represented languages. The large number of accepted papers and Findings papers explore a diverse set of methods, from meta-learning to tokenization and instruction tuning. In addition, shared task on multilingual multi-task information retrieval provided new data for NER and QA for a typologically diverse set of languages. The workshop day is jam-packed with excellent invited talks, poster, shared task and best paper sessions.\nUnanswerability and attribution in QA\nIn question answering (QA), a crucial challenge for current LLMs is hallucinating answers. A scenario where such hallucinations are common is when questions do not have an answer. To deal with hallucinations, a promising strategy is to train the model to attribute the answer to relevant references.\nThe Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models (Slobodkin et al.)\n.\nThis work shows that LLMs are aware of the concept of (un)answerability and that the representation of the first decoded token provides a strong indicator whether a question is answerable (removing this information in the first token significantly decreases performance). Furthermore, mentioning that the question is unanswerable in the prompt improves performance.\nThese results highlight that developing better decoding methods will also help make LLMs more factual.\nEvaluating and Modeling Attribution for Cross-Lingual Question Answering (Muller et al.)\n*.\nThis paper introduces attribution for cross-lingual question answering where the document supporting the generated answer may be in a different language than the question and answer. It creates the\nXOR-AttriQA dataset\nto measure attribution of SOTA QA models across 5 languages. Surprisingly, a large portion of generated answers are not attributable to any retrieved passage (up to 47% of\ncorrectly\npredicted answers in Japanese are not attributable).\nCurrent QA systems are thus often right but without any evidence, making them untrustworthy.\nMultilingual LLMs can be used to accurately detect attribution (which can complement string-based evaluation metrics) and can be used to rerank generated answers, improving QA performance.\nKey research directions are a) improving retrieval of cross-lingual passages and b) designing robust LLM-based metrics for QA evaluation.\nInstruction tuning\nInstruction tuning is a common way to improve LLMs for downstream settings and to align them to human behavior. However, current instruction tuning datasets still have their limitations (see\nthis post\nfor an overview).\nThe CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning (Kim et al.).\nThis paper introduces the Chain-of-Thought (CoT) Collection, which augments the\nFlan collection\ncovering 1,060 tasks with 1.84M chain-of-thought rationales. This amount of chain-of-thought instruction tuning data is particularly useful for smaller LMs and improves their performance on reasoning tasks including BIG-bench Hard and the multilingual MGSM benchmark.\nTask Adaptation\nWhile LLMs achieve very strong performance in a zero-shot setting, it is necessary to fine-tune them on task data to achieve the best performance. Keeping models updated as the distribution changes and encoding task knowledge efficiently across many settings are key challenges in this area.\nMeta-Learning Online Adaptation of Language Models (Hu et al.)\n.\nKeeping LLMs up-to-date is an important challenge as it is prohibitive to re-train these models. This paper hypothesizes that when continual fine-tuning a model on a stream of documents, the learning signal of important documents may be drowned out. To ameliorate this, the authors propose to meta-train a small model to reweigh the LM loss for each token during online fine-tuning in order to maximize the QA model’s performance after a single weighted update. They find that this dynamic weighting significantly outperforms standard fine-tuning and weighting heuristics.\nAdapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning (Poth et al.)*.\nFull fine-tuning of LLMs has become prohibitive and requires parameter-efficient methods instead. This demo paper presents\nAdapters\n, a library for\nparameter-efficient and modular learning\nwith LLMs and the successor to\nadapter-transformers\n. Adapters integrates 10 diverse modular methods such as\nprompt tuning\n,\nprefix tuning\n,\nCompacter\n,\nLoRA\n, and\n(IA)³\ninto 20 state-of-the-art models for NLP, vision, and multi-modal applications.\nIt supports a range of operations on these modules such as grouping, stacking, fusing, splitting, and parallelizing, among others, which enable a variety of modeling approaches and research directions.\nOutlier Dimensions Encode Task-Specific Knowledge (Rudman et al.).\nThis paper shows that outlier dimensions (dimensions with a variance that is significantly higher than the average) in LLMs persist during fine-tuning. They also find that just using the embedding value of such a high-variance dimension with a linear threshold can achieve performance similar to using the full model for some tasks and models.\nWe already know that LLMs capture task knowledge in a low-dimensional subspace (see\nAghajanyan et al., 2021\n, for instance)—but the observation that the subspace can be 1D for some settings can motivate the development of new efficient methods.\nNLG Evaluation\nAs LLMs are increasingly applied to generate natural language text, we need better metrics to evaluate their performance. One of the most promising directions is to use LLMs themselves as part of the metric, whether in a zero-shot setting or fine-tuned on relevant data.\nG-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (Liu et al.).\nThis paper proposes G-Eval, a framework for NLG evaluation using LLMs as reference-free metrics. Given a description of the task and the evaluation criteria, they first generate a more detailed CoT-style description of the evaluation steps using an LLM. All descriptions are then concatenated with the input example and fed to the LLM. Rather than directly predicting a score for each evaluation criterion, the authors observed that they obtain better measurements if they instead take the sum of all candidate scores weighted with their probability. On summarization, the framework achieves a higher correlation with human judgements than existing metrics.\nINSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback (Xu et al.).\nThis paper proposes InstructScore, a fine-grained reference-based explainable metric for NLG evaluation using LLMs. To fine-tune the LLM as the metric, they first collect unlabeled sentences. They then specify the number of errors, error types, and their severity labels for each sentence and ask GPT-4 to generate an incorrect sentence containing the errors matching the criteria and an explanation for each error. LLaMa is then fine-tuned on the generated data to identify and explain the errors in the incorrect sentence compared to the reference. LLaMa is further refined using feedback from GPT-4 regarding the correctness of the generated explanations. In practice, InstructScore achieved similar or higher correlation with human judgements than existing metrics on translation and NLG tasks.\nMultilingual Models\nWhile current LLMs excel on many tasks for English, performance is still much worse on languages with limited data. We thus require models that perform well for such languages and new methods to effectively scale models to these languages.\nFinGPT: Large Generative Models for a Small Language (Luukkonen et al.).\nThis paper is a comprehensive study of training LLMs for a small language (Finnish) including the collection of a diverse dataset, monolingual training at different model sizes (up to 13B parameters), adaptation of an existing multilingual language model (BLOOM) to the new language, and creation of a language-specific benchmark. The trained models outperform all previous models for Finnish while the language-adapted multilingual model outperforms the monolingual models.\nOverall, this is a nice blueprint of how LLMs can be trained for medium-resource languages.\nmmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations (Pfeiffer et al.)\n*†\n.\nThis paper proposes mmT5, the first modular multilingual generative model. The mT5-style model is pre-trained with language-specific modules and dramatically outperforms mT5 and at similar parameter sizes while matching or outperforming XLM-R.\nImportantly, the model’s modularity enables more direct control over its outputs.\nWhile mT5 generates text in the correct language in only 7% of cases for zero-shot cross-lingual summarization, mmT5 generates text in the correct language in 99% (!) of cases.\nXLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models (Liang et al.).\nThis paper proposes XLM-V, an XLM-R-style model covering 100 languages that is pre-trained with a 1M vocabulary. To create the vocabulary, vocabularies of languages are first clustered (\nChung et al., 2020\n), clusters are allocated capacity corresponding to their average log probability (\nZheng et al., 2021\n), and sentencepiece models are trained for each cluster and then combined. While pre-training with a 1M vocabulary is 2.5x slower than with a 250k vocabulary, the resulting model outperforms a (reimplemented) XLM-R.\nRomanization-based Large-scale Adaptation of Multilingual Language Models (Purkayastha et al.)\n*†\n.\nThis paper explores the potential of large-scale transliteration to enable multilingual LMs to deal with under-represented languages. In particular, the paper romanizes (i.e., maps UTF-8 characters to Latin characters) text using\nuroman\nacross 14 diverse languages, which is then used to adapt multilingual LMs. Romanization is particularly useful in the most challenging setups: on languages with unseen scripts and with limited training data.\nMultilingual Datasets and Evaluation\nA key challenge for multilingual NLP is the lack of evaluation datasets and studies that accurately assess the performance of multilingual models. The creation of new datasets and the development of new evaluation measures and analyses is thus an important research direction.\nMultilingual Large Language Models Are Not (Yet) Code-Switchers (Zhang et al.).\nThis paper evaluates LLMs on three code-switching tasks: sentiment analysis (English-{Spanish, Malayalam, Tamil}), translation (English-Hindi), and word-level language identification (English-Hindi, Standard Arabic-Egyptian Arabic). They observe that smaller fine-tuned multilingual LMs (XLM-R, mDeBERTa) still outperform zero-shot prompted LLMs on these tasks.\nXTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages (Ruder et al.)*†\n.\nXTREME-UP is a new benchmark focusing on user-centric tasks in under-represented languages with realistic amounts of available data. The benchmark includes impactful multi-modal tasks such as ASR and OCR, which we make accessible for text-only models by providing baseline system outputs (in addition to the original audio and image inputs). We created new data for a range of different tasks and updated standard tasks such as QA and NER to make them more practically relevant. We find that multilingual fine-tuned models still outperform few-shot prompted models on most tasks and that character-level modeling is beneficial. Overall, there is still a lot of headroom left to improve performance on under-represented languages.\nThe Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages (Zhang et al.).\nThis paper introduces SPARROW, a multilingual multi-task benchmark spanning 169 datasets from different online platforms to measure\nsociopragmatic\nunderstanding in LLMs (i.e., how well they perform on tasks related to social interactions such as sentiment analysis, emotion detection, etc). They observe that the fine-tuned models outperform zero-shot prompted models as well as ChatGPT. LLMs perform particularly poorly on humor and antisocial language detection and ChatGPT performs poorly across most languages in comparison to the best model.\nAfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages (Muhammad et al.)*.\nThis paper introduces AfriSenti, a sentiment analysis benchmark consisting of 110k+ tweets in 14 African languages. The dataset was used in the\nAfriSenti SemEval-2023 Shared Task\n. Data collection and annotation challenges included a lack of support for African languages by the Twitter API, lack of tone markings, frequent code-mixing and dialects, sarcasm and ambiguities, and a lack of annotators and a reliable Internet connection. The strongest model,\nAfroXLM-R\n, achieves 67.2 accuracy across all languages, leaving ample room for improvement.\nLarge Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU (Koto et al.).\nThis paper introduces IndoMMLU, the first benchmark on Indonesian language and culture consisting of 15k questions from primary school to university entrance exams. Among the 24 evaluated models, GPT-3.5 is the only one that passes primary school exams while no LLM demonstrates familiarity with local Indonesian languages and culture. The language exams also enable assessing the level of Indonesian language proficiency. For grades 7 and above, GPT-3.5 fails to pass the exam while other models only pass grades 1–3.\nTaTA: A Multilingual Table-to-Text Dataset for African Languages (Gehrmann et al.)*†.\nThis paper proposes\nTable-to-Text in African languages (TaTA)\n, the first large multilingual table-to-text dataset with a focus on African languages. TaTA was created by transcribing figures and associated text in bilingual reports by the\nDHS Program\n, which were then professionally translated to make the dataset fully parallel. We find that less than half of the outputs from an mT5-XXL-based model are understandable and attributable to the source data. We also observe that existing metrics perform poorly for multilingual table-to-text generation and introduce a new learned metric that achieves a high correlation with human judgements.", "metadata": { "title": "EMNLP 2023 Primer", "description": "An overview of EMNLP 2023 papers covering QA, instruction tuning, task adaptation, NLG evaluation, and multilingual models and datasets.", "keywords": null, "author": null, "og:site_name": "ruder.io", "og:type": "article", "og:title": "EMNLP 2023 Primer", "og:description": "An overview of EMNLP 2023 papers covering QA, instruction tuning, task adaptation, NLG evaluation, and multilingual models and datasets.", "og:url": "https://www.ruder.io/emnlp-2023-primer/", "og:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2024/10/xtreme-up_data.png", "og:image:width": "1200", "og:image:height": "393", "twitter:card": "summary_large_image", "twitter:title": "EMNLP 2023 Primer", "twitter:description": "An overview of EMNLP 2023 papers covering QA, instruction tuning, task adaptation, NLG evaluation, and multilingual models and datasets.", "twitter:url": "https://www.ruder.io/emnlp-2023-primer/", "twitter:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2024/10/xtreme-up_data.png", "twitter:label1": "Written by", "twitter:data1": "Sebastian Ruder", "twitter:site": "@seb_ruder", "article:published_time": "2023-12-05T20:29:00.000Z", "article:modified_time": "2024-10-23T13:19:55.000Z" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2024/10/xtreme-up_data.png", "bytes": 204976, "reused": true, "local_path": "attachments/d3fbf4449afb/images/001.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2024/10/xtreme-up_data.png", "alt": "EMNLP 2023 Primer", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://www.ruder.io/author/sebastian/", "label": "", "type": "a" }, { "href": "https://aclanthology.org/2023.findings-emnlp.125/", "label": "Ruder et al., 2023", "type": "a" }, { "href": "https://2023.emnlp.org/?ref=ruder.io", "label": "EMNLP 2023", "type": "a" }, { "href": "https://www.ruder.io/acl-2021-highlights/", "label": "earlier years", "type": "a" }, { "href": "https://arxiv.org/abs/2201.11903?ref=ruder.io", "label": "chain-of-thought prompting", "type": "a" }, { "href": "https://www.ruder.io/neurips-2023-primer/", "label": "such papers were more common", "type": "a" }, { "href": "https://newsletter.ruder.io/p/emnlp-2023-primer?ref=ruder.io", "label": "NLP News", "type": "a" }, { "href": "https://genbench.org/workshop/?ref=ruder.io", "label": "GenBench Workshop", "type": "a" }, { "href": "https://github.com/google/BIG-bench?ref=ruder.io", "label": "BIG-bench", "type": "a" }, { "href": "https://genbench.org/cbt/?ref=ruder.io", "label": "collaborative benchmarking tasks", "type": "a" }, { "href": "https://genbench.org/workshop_programme/?ref=ruder.io", "label": "program", "type": "a" }, { "href": "https://nlposs.github.io/2023/index.html?ref=ruder.io", "label": "Workshop for NLP Open-Source Software (NLP-OSS)", "type": "a" }, { "href": "https://github.com/CarperAI/trlx?ref=ruder.io", "label": "trlX", "type": "a" }, { "href": "https://github.com/aisingapore/sealion?ref=ruder.io", "label": "SEA-LION", "type": "a" }, { "href": "https://www.bigpictureworkshop.com/?ref=ruder.io", "label": "The Big Picture Workshop: Crafting a Research Narrative", "type": "a" }, { "href": "https://www.bigpictureworkshop.com/accepted-papers?ref=ruder.io", "label": "diverse set of accepted papers", "type": "a" }, { "href": "https://sigtyp.github.io/ws2023-mrl.html?ref=ruder.io", "label": "Multilingual Representation Learning Workshop (MRL)", "type": "a" }, { "href": "https://arxiv.org/abs/2310.11877?ref=ruder.io", "label": "The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Languag", "type": "a" }, { "href": "https://arxiv.org/abs/2305.14332?ref=ruder.io", "label": "Evaluating and Modeling Attribution for Cross-Lingual Question Answering (Muller et al.)", "type": "a" }, { "href": "https://github.com/google-research/google-research/tree/master/xor_attriqa?ref=ruder.io", "label": "XOR-AttriQA dataset", "type": "a" }, { "href": "https://www.ruder.io/an-overview-of-instruction-tuning-data/", "label": "this post", "type": "a" }, { "href": "https://arxiv.org/abs/2305.14045?ref=ruder.io", "label": "The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning (Kim e", "type": "a" }, { "href": "https://arxiv.org/abs/2301.13688?ref=ruder.io", "label": "Flan collection", "type": "a" }, { "href": "https://arxiv.org/abs/2305.15076?ref=ruder.io", "label": "Meta-Learning Online Adaptation of Language Models (Hu et al.)", "type": "a" }, { "href": "https://arxiv.org/abs/2311.11077?ref=ruder.io", "label": "Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning (Poth et al.)*.", "type": "a" }, { "href": "https://github.com/adapter-hub/adapters?ref=ruder.io", "label": "Adapters", "type": "a" }, { "href": "https://www.modulardeeplearning.com/?ref=ruder.io", "label": "parameter-efficient and modular learning", "type": "a" }, { "href": "https://arxiv.org/abs/2104.08691?ref=ruder.io", "label": "prompt tuning", "type": "a" }, { "href": "https://arxiv.org/abs/2101.00190?ref=ruder.io", "label": "prefix tuning", "type": "a" }, { "href": "https://arxiv.org/abs/2106.04647?ref=ruder.io", "label": "Compacter", "type": "a" }, { "href": "https://arxiv.org/abs/2106.09685?ref=ruder.io", "label": "LoRA", "type": "a" }, { "href": "https://proceedings.neurips.cc/paper_files/paper/2022/file/0cde695b83bd186c1fd456302888454c-Paper-Conference.pdf?ref=ruder.io", "label": "(IA)³", "type": "a" }, { "href": "https://aclanthology.org/2021.acl-long.568/?ref=ruder.io", "label": "Aghajanyan et al., 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2303.16634?ref=ruder.io", "label": "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (Liu et al.).", "type": "a" }, { "href": "https://arxiv.org/abs/2305.14282?ref=ruder.io", "label": "INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback (Xu et al.).", "type": "a" }, { "href": "https://arxiv.org/abs/2311.05640?ref=ruder.io", "label": "FinGPT: Large Generative Models for a Small Language (Luukkonen et al.).", "type": "a" }, { "href": "https://arxiv.org/abs/2305.14224?ref=ruder.io", "label": "mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations (Pfeiffer et al.)", "type": "a" }, { "href": "https://arxiv.org/abs/2305.11938?ref=ruder.io", "label": "*†", "type": "a" }, { "href": "https://arxiv.org/abs/2301.10472?ref=ruder.io", "label": "XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models (Liang et al.).", "type": "a" }, { "href": "https://aclanthology.org/2020.emnlp-main.367/?ref=ruder.io", "label": "Chung et al., 2020", "type": "a" }, { "href": "https://aclanthology.org/2021.emnlp-main.257/?ref=ruder.io", "label": "Zheng et al., 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2304.08865?ref=ruder.io", "label": "Romanization-based Large-scale Adaptation of Multilingual Language Models (Purkayastha et al.)", "type": "a" }, { "href": "http://(%20utf-8%20characters%20to%20latin%20characters)%20text/", "label": "uroman", "type": "a" }, { "href": "https://arxiv.org/abs/2305.14235?ref=ruder.io", "label": "Multilingual Large Language Models Are Not (Yet) Code-Switchers (Zhang et al.).", "type": "a" }, { "href": "https://arxiv.org/abs/2310.14557?ref=ruder.io", "label": "The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages (Zhang et al.).", "type": "a" }, { "href": "https://arxiv.org/abs/2302.08956?ref=ruder.io", "label": "AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages (Muhammad et al.)*.", "type": "a" }, { "href": "https://afrisenti-semeval.github.io/?ref=ruder.io", "label": "AfriSenti SemEval-2023 Shared Task", "type": "a" }, { "href": "https://aclanthology.org/2022.coling-1.382/?ref=ruder.io", "label": "AfroXLM-R", "type": "a" }, { "href": "https://arxiv.org/abs/2310.04928?ref=ruder.io", "label": "Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU (Koto et al.).", "type": "a" }, { "href": "https://arxiv.org/abs/2211.00142?ref=ruder.io", "label": "TaTA: A Multilingual Table-to-Text Dataset for African Languages (Gehrmann et al.)*†.", "type": "a" }, { "href": "https://github.com/google-research/url-nlp/tree/main/tata?ref=ruder.io", "label": "Table-to-Text in African languages (TaTA)", "type": "a" }, { "href": "https://dhsprogram.com/?ref=ruder.io", "label": "DHS Program", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:52:05Z", "source_url": "https://www.ruder.io/emnlp-2023-primer/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/2201.11903", "bytes": 891773, "reused": true, "local_path": "papers/repair/52cc4c4f6cf368123b97.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2310.11877", "bytes": 1229951, "reused": false, "local_path": "papers/repair/17eb9a8119efc9e53e89.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2305.14332", "bytes": 1640568, "reused": false, "local_path": "papers/repair/3e02fa3e5f74012f5ebc.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2305.14045", "bytes": 4555199, "reused": false, "local_path": "papers/repair/a13de1eca9251f3c19ed.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2301.13688", "bytes": 899298, "reused": true, "local_path": "papers/repair/5f6eb2980131e507aabc.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2305.15076", "bytes": 1390238, "reused": false, "local_path": "papers/repair/315da631ac479874cd56.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2311.11077", "bytes": 1475670, "reused": false, "local_path": "papers/repair/e9f54180d623083531c5.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2104.08691", "bytes": 547768, "reused": true, "local_path": "papers/repair/2ac4e46403c986515128.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2101.00190", "bytes": 1569113, "reused": true, "local_path": "papers/repair/a704430cffe77134ac6b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2106.04647", "bytes": 721967, "reused": false, "local_path": "papers/repair/73201fff20832aee5884.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2106.09685", "bytes": 1609513, "reused": true, "local_path": "papers/repair/e7d2fed957472d7677ec.pdf" }, { "status": "ok", "url": "https://proceedings.neurips.cc/paper_files/paper/2022/file/0cde695b83bd186c1fd456302888454c-Paper-Conference.pdf?ref=ruder.io", "bytes": 452073, "reused": false, "local_path": "papers/repair/f6ee21d13b9c1cf5a511.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2303.16634", "bytes": 415272, "reused": false, "local_path": "papers/repair/c4a7db9d4275f605e432.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2305.14282", "bytes": 960188, "reused": false, "local_path": "papers/repair/483b06e683fdcc945a80.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2311.05640", "bytes": 613149, "reused": false, "local_path": "papers/repair/428f401d51ccda7f7f77.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2305.14224", "bytes": 3046506, "reused": false, "local_path": "papers/repair/9816bf112bdf55182098.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2305.11938", "bytes": 1383799, "reused": false, "local_path": "papers/repair/b68a482784883537d850.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2301.10472", "bytes": 547552, "reused": false, "local_path": "papers/repair/95b1c896a33309e9ae6f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2304.08865", "bytes": 1373270, "reused": false, "local_path": "papers/repair/20cdc3de7ce8c8327ca8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2305.14235", "bytes": 2028430, "reused": false, "local_path": "papers/repair/c905429a6ad6a95dd0f4.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2310.14557", "bytes": 1500404, "reused": false, "local_path": "papers/repair/b081fc3183978087527e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2302.08956", "bytes": 1269395, "reused": false, "local_path": "papers/repair/38b3e22856350c5130c8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2310.04928", "bytes": 1093933, "reused": false, "local_path": "papers/repair/85b9999602d77a36d974.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2211.00142", "bytes": 1565744, "reused": false, "local_path": "papers/repair/f48c47089a3683e96fde.pdf" } ], "actual_video_files": [], "body_chars": 18163, "expected_body_images": 1, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/d3fbf4449afb/pages/source.html", "local_html": "attachments/d3fbf4449afb/pages/article.html", "body_base_path": "attachments/d3fbf4449afb/pages/", "body_markdown_path": "attachments/d3fbf4449afb/pages/article.md" }