{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Sebastian Ruder",
  "title_listed": "Multi-domain Multilingual Question Answering",
  "keywords": "多语言,问答,跨领域",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.ruder.io/multi-qa-tutorial/",
  "publish_date": "2021-12",
  "corpus_stage": "预训练/SFT",
  "notes": "多领域多语言问答综述（EMNLP教程）",
  "id": "e29f1e8231ed",
  "collected_at": "2026-10-02T14:16:23",
  "title": "Multi-domain Multilingual Question Answering",
  "body_markdown": "Multi-domain Multilingual Question Answering\nThis post expands on the EMNLP 2021 tutorial on Multi-domain Multilingual Question Answering and highlights key insights and takeaways.\nSebastian Ruder\n06 Dec 2021\n•\n21 min read\nShare\nThis post expands on the\nEMNLP 2021 tutorial on Multi-domain Multilingual Question Answering\n.\nThe tutorial was organised by\nAvi Sil\nand me. In this post, I highlight key insights and takeaways of the tutorial. The slides are\navailable online\n. You can find the table of contents below:\nIntroduction\nOpen-Retrieval QA vs Reading Comprehension\nWhat is a Domain?\nMulti-Domain QA\nDatasets for Multi-Domain QA\nMulti-Domain QA Models\nUnsupervised Domain Adaptation for QA\nDomain Adaptation with Pre-trained LMs\nDomain Generalization\nMultilingual QA\nDatasets for Multilingual QA\nIssues of Multilingual QA Datasets\nCreating QA Datasets\nMultilingual QA Evaluation\nMultilingual QA Models\nModels for Multilingual Reading Comprehension\nModels for Multilingual Open-Retrieval QA\nOpen Research Directions\nIntroduction\nQuestion answering is one of the most impactful tasks in natural language processing (NLP). In the tutorial, we focus on two main categories of question answering studied in the literature: open-retrieval question answering (ORQA) and reading comprehension (RC).\nOpen-Retrieval QA vs Reading Comprehension\nOpen-retrieval QA focuses on the most general setting where given a question we first need to retrieve relevant documents from a large corpus such as Wikipedia. We then process these documents to identify the relevant answer as can be seen below. We avoid using the term open-domain QA as \"open-domain\" may also refer to a setting covering many domains.\nOpen-Retrieval Question Answering (image based on\nChen et al., 2017\n)\nReading comprehension can be seen as a sub-problem of open-retrieval QA as it assumes that we have access to the gold paragraph that contains the answer (see below). We then only need to find the corresponding answer in this paragraph. In both settings, answers are commonly represented as a minimal span.\nReading Comprehension assumes a gold paragraph is provided\nStandard approaches for reading comprehension build on pre-trained models such as BERT. The model is provided with the question and candidate paragraph as input and is trained to predict whether the question is answerable (typically using the representation associated with its [CLS] token) and whether each token is the start or end of an answer span, which can be seen below. The same approach can be used for ORQA, with some modifications (\nAlberti et al., 2019\n).\nUsing BERT for reading comprehension involves fine-tuning it to predict a) whether a question is answerable and b) whether each token is the start and end of an answer span.\nInformation retrieval (IR) methods are used to retrieve the relevant paragraphs. Classic sparse methods such as BM25 (\nRobertson et al., 2009\n) do not require any training as they weigh terms and documents based on their frequency using\ntf-idf\nmeasures. Recent dense neural approaches such as DPR (\nKarpukhin et al., 2020\n) train models to maximize the similarity between question and passage and then retrieve the most relevant passages via maximum inner product search.\nWhat is a Domain?\nA domain can be seen as a manifold in a high-dimensional variety space consisting of many dimensions such as socio-demographics, language, genre, sentence type, etc (\nPlank et al., 2016\n). Domains differ in terms of their granularity and large domains such as Twitter can consist of several more narrow domains to which we might want to adapt our models. Two major facets of this variety space are\ngenre\n(we will use this term interchangeably with 'domain') and\nlanguage\n, which will be the focus of this post.\nDomain granularity (Credit:\nGururangan et al., 2020\n)\nMulti-Domain QA\nIn each of the two main sections of this post, we will first discuss common datasets and then modelling approaches.\nDatasets for Multi-Domain QA\nResearch in question answering has spanned many domains, as can be seen below. The most common domain is\nEncyclopedia\n, which covers many Wikipedia-based datasets such as SQuAD (\nRajpurkar et al., 2016\n), Natural Questions (NQ;\nKwiatkowski et al., 2019\n), DROP (\nDua et al., 2019\n), and WikiReading (\nHewlett et al., 2016\n), among many others. Datasets in this domain are typically referred to as \"open-domain\" QA.\nRC Olympics: The many domains of reading comprehension\nDatasets in the\nFiction\ndomain typically require processing narratives in books such as NarrativeQA (\nKočiský et al., 2018\n), Children's Book Test (\nHill et al., 2016\n), and BookTest (\nBajgar et al., 2016\n) or narratives written by crowd workers such as MCTest (\nRichardson et al., 2013\n), MCScript (\nModi et al., 2016\n), and ROCStories (\nMostafazedh et al., 2016\n).\nAcademic tests\ndatasets target science questions in US school tests such as ARC (\nClark et al., 2018\n), college-level exam resources such as ReClor (\nYu et al., 2019\n) and school-level exam resources such as RACE (\nLai et al., 2017\n).\nNews\ndatasets include NewsQA (\nTrischler et al., 2017\n), CNN / Daily Mail (\nHermann et al., 2015\n), and NLQuAD (\nSoleimani et al., 2021\n).\nIn addition, there are datasets focused on\nSpecialized Expert Materials\nincluding manuals, reports, scientific papers, etc. Such domains are most common in industry as domain-specific chatbots are increasingly used by companies to respond to users queries but associated datasets are rarely made available. Existing datasets focus on tech forums such as TechQA (\nCastelli et al., 2020\n) and AskUbuntu (\ndos Santos et al., 2015\n) as well as scientific articles including BioASQ (\nTsatsaronis et al., 2015\n) and Qasper (\nDasigi et al., 2021\n). In addition, the pandemic saw the creation of many datasets related to COVID-19 such as COVID-QA-2019 (\nMöller et al., 2020\n), COVID-QA-147 (\nTang et al., 2020\n), and COVID-QA-111 (\nLee et al., 2020\n).\nBeyond these datasets focusing on individual domains, there are several datasets spanning multiple domains such as CoQA (\nReddy et al., 2019\n), QuAIL (\nRogers et al., 2020\n), and MultiReQA (\nGuo et al., 2021\n), among others. Lastly, the\nORB\nevaluation server (\nDua et al., 2019\n) enables the evaluation of systems across datasets spanning multiple domains.\nMulti-Domain QA Models\nIn most cases when learning in a multi-domain setting there may be limited or no labelled data available in the target domain available. We discuss how to adapt to a target domain using only unlabelled data, the most effective way to use pre-trained language models for domain adaptation in QA, and how to generalize across domains.\nUnsupervised Domain Adaptation for QA\nUnsupervised domain adaptation for QA assumes access to labelled data in a source domain and unlabelled target domain data. In the absence of labelled gold data in the target domain, most methods rely on generating 'silver' <question, paragraph, answer> data in the target domain. To this end, these methods train a question generation model based on a pre-trained LM on the source domain and then apply it to the target domain to generate synthetic questions given answer spans (\nWang et al., 2019\n;\nShakeri et al., 2020\n;\nYue et al., 2021\n). The QA model is then trained jointly on the gold source domain and silver target domain data, as can be seen below. This is often combined with other domain adaptation strategies such as adversarial learning and self-training (\nCao et al., 2020\n).\nJoint training on gold source and silver target domain data that is generated using BART or T5\nDomain Adaptation with Pre-trained LMs\nIn many scenarios with specialized domains, a general pre-trained language model such as BERT may not be sufficient. Instead,\ndomain-adaptive fine-tuning\n(\nHoward & Ruder, 2018\n;\nGururangan et al., 2020\n) of a pre-trained LM on target domain data typically performs better. Recent examples such as BioBERT (\nLee et al., 2019\n) and SciBERT (\nBeltagy et al., 2019\n) have been effective for biomedical and scientific domains respectively.\nWhen labelled data in a source domain and in the target domain are available, a general recipe is to first fine-tune a model on labelled source domain data and then to subsequently fine-tune it on labelled data in the target domain. However, naive fine-tuning generally leads to deterioration of performance in the source domain, which may be undesirable. Instead, strategies from continual learning such as L2 regularization (\nXu et al., 2019\n) ensure that the parameters of the model fine-tuned on the target domain do not diverge significantly from the source domain model, thereby reducing catastrophic forgetting as can be seen below.\nStrategies such as L2 regularization result in reasonable performance on the source domain (SQuAD) after fine-tuning on the target domain (BioASQ)\nFor very low-resource domains, another strategy is to explicitly adapt the model to characteristics of the target domain. If the target domain contains specialized terms, the model's vocabulary can be augmented with these terms to learn better representations. In addition, for many domains, the structure of the unlabelled data can contain information that may be useful for the end task. This structure such as the abstract, error description, associated cause, etc can be leveraged to generate synthetic data on which the model can be fine-tuned (\nZhang et al., 2020\n).\nIn the open-retrieval setting, dense retrieval methods (\nKarpukhin et al., 2020\n) trained on a source domain may not generalize in a zero-shot manner to low-resource target domains. Instead, we can leverage the same question generation methods discussed in the previous section to create silver data for training a retrieval model on the target domain (\nReddy et al., 2021\n). Ensembling over BM25 and the adapted DPR model yields the best results.\nDomain Generalization\nIn practice, pre-trained models fine-tuned on a single domain often generalize poorly. Training on multiple source distributions reduces the need for selecting a single source dataset (\nTalmor & Berant, 2019\n). Additional fine-tuning on a related task helps even when using pre-trained models.\nHowever, different datasets often have different formats, which makes it difficult to train a joint model for them without task-specific engineering. Recent pre-trained LMs such as T5 facilitate such cross-dataset learning as each dataset can be converted to a unified text-to-text format. Training a model jointly across multiple QA datasets with such a text-to-text format leads to better generalization to unseen domains (\nKhashabi et al., 2020\n).\nMultilingual QA\nLanguage can be seen as another facet of the domain manifold. As we will see, many methods discussed in the previous section can also be successfully applied to the multilingual setting. At the same time, multilingual QA poses its own unique challenges.\nDatasets for Multilingual QA\nMany early multilingual IR and QA datasets have been collected as part of community evaluations. Many test collections used newswire articles, e.g.\nCLIR at TREC\n1994–2004, QA at CLEF 2003–2005 (\nMagnini et al., 2003\n) or Wikipedia, e.g. QA at CLEF 2006–2008 (\nGillard et al., 2006\n). Such datasets focused mainly on Indo-European languages, though more recent ones also covered other languages, e.g. Hindi and Bengali at FIRE 2012 (\nYadav et al., 2012\n).\nThere are wide range of\nmonolingual reading comprehension\ndatasets\n, many of them variants of SQuAD (\nRajpurkar et al., 2016\n). Most of them are available in Chinese, Russian, and French and they generally consist of naturalistic data in each language.\nMonolingual open-retrieval QA\ndatasets\nare more diverse in nature. They differ based on the type and amount of context they provide and often focus on specialized domains, from Chinese history exams (\nZhao & Zhao, 2018\n) to Chinese maternity forums (\nXu et al., 2020\n) and Polish 'Did you know?' questions (\nMarcinczuk et al., 2013\n).\nThe MLQA alignment and annotation process (\nLewis et al., 2020\n)\nMultilingual reading comprehension\ndatasets\nhave generally been created using translations. MLQA (\nLewis et al., 2020\n), which can be seen above has been created by automatically aligning Wikipedia paragraphs across multiple languages and annotating questions and answers on the aligned paragraphs. Such automatic alignment, however, may lead to severe quality issues for some languages (\nCaswell et al., 2021\n) and may result in overfitting to the biases of the alignment model. Another dataset, XQuAD (\nArtetxe et al., 2020\n) was created by professionally translating a subset of SQuAD to 10 other languages. Finally, MLQA-R and XQuAD-R (\nRoy et al., 2020\n) are conversions of the former datasets to the answer sentence retrieval setting.\nMultilingual open-retrieval QA\ndatasets\ntypically consist of naturalistic data. XQA (\nLiu et al., 2019\n) covers 'Did you know?' Wikipedia questions converted to a\nCloze format\n. For each question, the top 10 Wikipedia documents ranked by BM25 are provided as context. TyDi QA (\nClark et al., 2020\n) ask annotators to write \"information-seeking\" questions based on short Wikipedia prompts in typologically diverse languages. Such information-seeking questions lead to less lexical overlap compared to RC datasets like MLQA and XQuAD and thus result in a more challenging QA setting. However, as in-language Wikipedias are used for finding context paragraphs (via Google Search) and as the Wikipedias of many under-represented languages are very small, many questions in TyDi QA are unanswerable.\nThe XOR-TyDi QA annotation process (\nAsai et al., 2021\n)\nXOR-TyDi QA (\nAsai et al., 2021\n) addresses this issue via the above procedure, which translates unanswerable TyDi QA questions to English and retrieves context paragraphs from English Wikipedia. This strategy significantly decreases the fraction of unanswerable questions. While XOR-TyDi QA focuses on\ncross-lingual\nretrieval, Mr. TyDi (\nZhang et al., 2021\n) augments TyDi QA with in-language documents for evaluating\nmonolingual\nretrieval models. As answers in TyDi QA are spans in a sentence, Gen-TyDi QA (\nMuller et al., 2021\n) extends the dataset with human-generated answers to enable the evaluation of generative QA models. Finally, MKQA (\nLongpre et al., 2020\n) translates 10k queries from Natural Questions (\nKwiatkowski et al., 2019\n) to 25 other languages. In addition, it augments the dataset with annotations that link directly against Wikidata entities, enabling evaluation beyond span extraction.\nAn emerging category of multilingual QA datasets is\nmultilingual common sense reasoning\n. Such datasets consist of multiple-choice assertions that are translated into other languages (\nPonti et al., 2020\n;\nLin et al., 2021\n).\nIssues of Multilingual QA Datasets\nExisting monolingual and multilingual QA datasets have some issues that one should be aware of.\nLanguage distribution\nExisting datasets predominantly focus on \"high-resource\" languages where large amounts of data are available. Evaluating QA models on such datasets provides a distorted view of progress in the field. For instance, questions that can be solved by string matching are easy in English but much harder in morphologically rich languages (\nClark et al., 2020\n). Among current key applications of NLP, QA has the lowest linguistic global utility, i.e. performance averaged across the world's languages (\nBlasi et al., 2021\n), which can be seen below. While QA datasets cover languages with many speakers, there is still a long way to go in terms of an equitable coverage across the world's languages.\nLinguistic and demographic utility of different NLP applications (\nBlasi et al., 2021\n)\nHomogeneity\nIn order to make collection scalable, multilingual datasets often collect data that covers the same question or similar topics across languages, thus missing out on language-specific nuances. In addition, it is often not feasible to do an in-depth error analysis for every language. The most common source of homogeneity is translation, which carries its own biases.\nLimitations of translation\n\"Translationese\" differs in many aspects from natural language (\nVolanksy et al., 2015\n). Translated questions often do not have answers in a target language Wikipedia (\nValentim et al., 2021\n). In addition, datasets created via translation inherit artefacts such as a large train-test overlap of answers in NQ (\nLewis et al., 2020\n) and translation also leads to new artefacts, e.g. in NLI when premise and hypothesis are translated separately (\nArtetxe et al., 2020\n). Finally, translated questions differ from the types of questions \"naturally\" asked by speakers of different languages, leading to an English and Western-centric bias.\nEnglish and Western-centric bias\nExamples in many QA datasets are biased towards questions asked by English speakers. Cultures differ in what types of questions are typically asked, e.g. speakers outside the US probably would not ask about famous American football or baseball players. In COPA (\nRoemmele et al., 2011\n), many referents have no language-specific terms in some languages, e.g. bowling ball, hamburger, lottery (\nPonti et al., 2020\n). Common sense knowledge, social norms, taboo topics, assessments of social distance, etc are also culture-dependent (\nThomas, 1983\n). Lastly, the common setting of training on English data leads to an overestimation of transfer performance on languages similar to English and underestimation on more distant languages.\nDependence on retrieval\nThe standard setting of identifying a minimal span for open-domain QA in the retrieved documents benefits extractive systems. It assumes there is a single gold paragraph providing the correct answer and does not consider information from other paragraphs or pages. For unanswerable questions, answers may often be found in other pages that were not retrieved (\nAsai & Choi, 2021\n).\nInformation scarcity\nTypical knowledge resources such as language-specific Wikipedias often do not contain the relevant information, particularly for under-represented languages. For such languages, datasets must necessarily be cross-lingual. In addition, some information is only available from other sources, e.g. IMDb, news articles, etc.\nDifficulty of multilingual comparison\nComparing a model's performance across different languages is difficult due to a range of factors such as different levels of question difficulty, different amounts and quality of monolingual data, impact of translationese, etc. Instead, it is better to perform system-level comparisons across languages.\nMonolingual vs multilingual QA datasets\nCreating multilingual QA datasets is expensive and thus often infeasible with academic budgets. In contrast, work on monolingual QA datasets is often perceived as \"niche\". Such work, however, is arguably much more important and impactful than incremental modelling advances, which are commonly accepted to conferences (\nRogers et al., 2021\n). In order to foster inclusivity and diversity in NLP, it is key to enable and reward such work, particularly for under-represented languages. Language-specific QA datasets can go beyond a \"replication\" of English work by, for instance, performing analyses of language-specific phenomena and extending or improving the QA setting.\nCreating QA Datasets\nEfficient multilingual QA evaluation at scale\nA key challenge of multilingual QA is the lack of data for many languages. Instead of labelling large amounts of data in every language in order to cover the entire distribution, we can create targeted tests that probe for specific capabilities, for instance using CheckList (\nRibeiro et al., 2020\n). This way, a small number of templates can cover many different model capabilities. Such template-based tests have so far been used for evaluating multilingual reading comprehension (\nRuder et al., 2021\n) and closed-book QA (\nJiang et al., 2020\n;\nKassner et al., 2021\n) where they enable fine-grained evaluation across languages as can be seen below. However, in order to scale such tests across languages native speaker expertise or translation are still required.\nError rate of mBERT (left) and XLM-R (right) fine-tuned on English SQuAD v1.1 across different languages in MultiCheckList (\nRuder et al., 2021\n)\nBest practices\nWhen creating a new QA dataset, it is important to focus on the research questions you want to answer with your dataset. Try to avoid creating confounding variables (translationese, morphology, syntax, etc) that obfuscate answering these questions. Consider collecting data in a typologically diverse set of languages and think about the use case of your dataset and how systems based on the data could help humans. Chose an appropriate dataset format: If you want to help people around the world answer questions, focus on information-seeking questions and avoid cultural bias. If you want to help users ask questions about a short document, focus on reading comprehension. Finally, in order to create inclusive and diverse QA datasets, it is important to work with speaker communities and conduct participatory research (\n∀ et al., 2020\n).\nMultilingual QA Evaluation\nCommon evaluation settings in multilingual QA range from\nmonolingual QA\nwhere all data is in the same language to\ncross-lingual\nscenarios where the question, context, and answer can be in different languages as well as\nzero-shot cross-lingual\ntransfer\nsettings where training data is in a high-resource language and test data is in another language.\nEvaluation metrics\nare based on lexical overlap using either Exact Match (EM) or mean token F1, with optional pre-processing of predictions and answers (\nLewis et al., 2020\n). Such token-based metrics, however, are not appropriate for languages without whitespace separation and require a language-specific segmentation method, which introduces a dependence on the evaluation setting. Furthermore, metrics based on string matching penalize morphologically rich languages as extracted spans may contain irrelevant morphemes, favour extractive over generative systems, and are biased towards short answers.\nAlternatively, evaluation can be performed on the character or byte level. As standard metrics used for natural language generation (NLG) such as BLEU or ROUGE show little correlation with human judgements for some languages (\nMuller et al., 2021\n), learned metrics based on strong pre-trained models such as BERTScore (\nZhang et al., 2020\n) or SAS (\nRisch et al., 2021\n) may be preferred, particularly for evaluating generative models.\nMultilingual QA Models\nMultilingual models for QA are generally based on pre-trained multilingual Transformers such as mBERT (\nDevlin et al., 2019\n), XLM-R (\nConneau et al., 2020\n), or mT5 (\nXue et al., 2021\n).\nModels for Multilingual Reading Comprehension\nFor reading comprehension, a multilingual model is typically fine-tuned on data in English and then applied to test data in the target language via zero-shot transfer. Recent models generally perform well on high-resource languages present in standard QA datasets while performance is slightly lower for languages with different scripts (\nRuder et al., 2021\n), which can be seen below. Fine-tuning the model on data in the target language using masked language modelling (MLM) before training on task-specific data generally improves performance (\nPfeiffer et al., 2020\n).\nZero-shot cross-lingual transfer performance (F1) of representative models on XQuAD (left) and MLQA (right) (\nRuder et al., 2021\n)\nIn practice, fine-tuning on a few labelled target language examples can significantly improve transfer performance compared to zero-shot transfer (\nHu et al., 2020\n;\nLauscher et al., 2020\n). However, the same does not hold for the more challenging open-retrieval QA (\nKirstain et al., 2021\n). Multi-task training on data in many languages improves performance further (\nDebnath et al., 2021\n).\nMost prior work on multilingual QA uses the\ntranslate-test\nsetting, which translates all data into English—typically using an online MT system as a black box—and then applies a QA model trained on English to it (\nHartrumpf et al., 2008\n;\nLin & Kuo, 2010\n;\nShima & Mitamura, 2010\n). In order to map the predicted English answer to the target language, back-translation of the answer span does not work well as it is agnostic of the paragraph context. Instead, recent methods employ the attention weights from a neural MT system to align the English answer span to a span in the original document (\nAsai et al., 2018\n;\nLewis et al., 2020\n).\nAlternatively, in the\ntranslate-train\nsetting the English training data is translated to the target language and a target language QA model is trained on the data. In this case, it is crucial to ensure that answer spans can be recovered after translation by enclosing them with tags or using fuzzy search (\nHsu et al., 2019\n;\nHu et al., 2020\n). We can go even further and translate the English data to\nall\ntarget languages on which we train a multilingual QA model. This\ntranslate-train-all\nsetting generally performs best for reading comprehension (\nConneau et al., 2020\n;\nHu et al., 2020\n;\nRuder et al., 2021\n) and achieves performance close to English on high-resource languages but lower on others. The below flowchart shows what method achieves the best performance depending on the available data.\nFlowchart for multilingual reading comprehension\nModels for Multilingual Open-Retrieval QA\nTranslate-test\nis the standard approach for open-retrieval QA as it only requires access to English resources. In addition to training a document reader model on English data, the open-retrieval setting also necessitates training an English retrieval model. During inference, we apply the models to the translated data and back-translate the answer to the target language, which can be seen below. However, low-quality MT may lead to error propagation and for some questions answers may not be available in the English Wikipedia. On the other hand,\ntranslate-train\nis generally infeasible in the open-retrieval setting as it requires translating all possible context (such as the entire Wikipedia) to the target language. As an alternative, only questions can be translated, which may outperform translate-test (\nLongpre et al., 2020\n).\nInference with translate-test in the open-retrieval QA setup (\nnote\n: flags are used as succinct visual representations and are not meant to reflect a particular language variety)\nWithout using translations, we need to train cross-lingual retrieval and document reader models that can assess similarity between questions and context paragraphs and answer spans respectively across languages. To this end, we need to fine-tune a pre-trained multilingual model using target language questions and English contexts. However, restricting retrieval to English documents limits the viewpoints and knowledge sources at our disposal. We would thus like to extend retrieval to documents in multiple languages.\nFine-tuning a pre-trained multilingual model to retrieve passages only on English data does not generalize well to other languages (\nZhang et al., 2021\n), similar to the multi-domain setting. In order to train a retrieval model that better generalizes to other languages, we can fine-tune the model on multilingual data instead (\nAsai et al., 2021\n). Alternatively, we can employ data augmentation. Similar to the multi-domain setting, we can obtain silver data in the target language by generating synthetic target language questions, in this case using a translate-train model (\nShi et al., 2021\n). In addition, we can obtain weakly supervised examples using language links in Wikipedia as can be seen below (\nShi et al., 2021\n;\nAsai et al., 2021\n). Specifically, we retrieve the articles corresponding to the original answer and answer paragraph in other languages and use them as new training examples. Lastly, a combination of BM25 + dense retrieval also performs best in this setting (\nZhang et al., 2021\n).\nCross-lingual data expansion via Wikidata language links; based on an example by\nAsai et al. (2021)\nTo aggregate the retrieved passages in different languages we can train a pre-trained multilingual text-to-text model such as mT5 to generate an answer when provided with the passages as input (\nMuller et al., 2021\n;\nAsai et al., 2021\n). The full pipeline consisting of multilingual retrieval and answer generation models can be seen below. As the model will only learn to generate answers in languages covered by existing datasets, data augmentation is again key. Furthermore, models can be iteratively trained using newly retrieved and newly identified answers as additional training data in subsequent iterations (\nAsai et al., 2021\n). The best models achieve strong performance in the full open-retrieval setting but there is still significant headroom left.\nMultilingual retrieval and multilingual answer generation pipeline (\nAsai et al., 2021\n)\nTwo of the most challenging aspects of multilingual open-retrieval QA are finding the paragraph containing the answer (\nparagraph selection\n) and identifying whether a document contains the answer to a query (\nanswerability prediction\n;\nAsai & Choi, 2021\n). A related problem is answer sentence selection where models predict whether a sentence contains the answer (\nGarg et al., 2020\n). Unanswerability is often due to errors in document retrieval or unanswerable questions requiring multiple paragraphs to answer. To address this headroom,\nAsai and Choi (2021)\nrecommend to a) go beyond using Wikipedia for retrieval; b) to improve the quality of annotated questions in existing and future datasets; and c) to move from extracting a span to generating the answer.\nOpen Research Directions\nMulti-modal question answering\nFor many language varieties and domains, it may be easier to obtain data in other modalities. As a recent example, SD-QA (\nFaisal et al., 2021\n) augments TyDi QA with spoken utterances matching the questions in four languages and multiple dialects.\nOther domains: time and geography\nA domain can include many facets not covered in existing work. For instance, answers often depend on extra-linguistic context such as the time and location where the questions were asked. SituatedQA (\nZhang & Choi, 2021\n) augments context-dependent questions in Natural Questions with time and geography-dependent contexts to study such questions.\nTemporal and geographical question contexts in SituatedQA (\nZhang & Choi, 2021\n)\nCode-switching\nCode-switching is a common phenomenon in multilingual communities but mostly neglected in QA research. Only few resources exist in Bengali, Hindi, Telugu, and Tamil (\nRaghavi et al., 2015\n;\nBanerjee et al., 2016\n;\nChandu et al., 2018\n;\nGupta et al., 2018\n). For a broader overview of code-switching, have a look at this survey (\nDoğruöz et al., 2021\n).\nMultilingual multi-domain generalization\nMost open-retrieval QA datasets only cover Wikipedia while many domains important in real-world applications (e.g. tech questions) only have English QA datasets. Other domains without much data are particularly relevant in non-Western contexts, e.g. finance for small businesses, legal and health questions. In addition, currently unanswerable questions require retrieving information from a wider set of domains such as IMDb (\nAsai et al., 2021\n). In order to create truly open-domain QA systems, we thus need to train open-retrieval QA systems to answer questions from many different domains.\nData augmentation\nGeneration of synthetic multilingual QA data has been little explored beyond translation and retrieval (\nShi et al., 2021\n). The generation of data about non-Western entities may be particularly helpful.\nGenerative question answering\nIn most existing QA datasets, the short answer is a span in the context. In order to train and evaluate models more effectively, more datasets need to include longer, more natural answers. The generation of long-form answers is particularly challenging, however (\nKrishna et al., 2021\n).\nAggregating evidence from diverse sources\nWe need to develop better aggregation methods that cover reasoning paths, e.g. for multi-hope reasoning (\nAsai et al., 2020\n). Models also need to be able to generate answers that are faithful to the retrieved passages, requiring clear answer attribution. Finally, we require methods that can effectively combine evidence from different domains and even different modalities.\nConversational question answering\nCurrent open-retrieval QA datasets are generally single-turn and do not depend on any external context. In order to train generally useful QA systems, models should also be able to take into account conversational context as required by datasets such as QuAC (\nChoi et al., 2018\n). In particular, they should be able to handle coreference, ask for clarification regarding ambiguous questions, etc.\nFurther reading\nHere are some additional resources that may be useful to learn more about different aspects of the topic:\na survey on\nquestion answering datasets\nwith a particular focus on the required reasoning skills (\nRogers et al., 2021\n);\na survey on\nneural unsupervised domain adaptation\nin NLP (\nRamponi & Plank, 2020\n);\nthe\nACL 2020 tutorial\non\nopen-domain question answering\n;\nand my\nACL 2019 tutorial\non\ncross-lingual representation learning\n.\nCredit\nThanks to the following people for feedback on a draft of the tutorial slides: Jon Clark, Tim Möller, Sara Rosenthal, Md Arafat Sultan, and Benjamin Muller.\nCitation\nIf you found this post helpful, consider citing\nthe tutorial\nas:\n@inproceedings{ruder-sil-2021-multi,\n    title = \"Multi-Domain Multilingual Question Answering\",\n    author = \"Ruder, Sebastian  and\n      Sil, Avi\",\n    booktitle = \"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts\",\n    month = nov,\n    year = \"2021\",\n    address = \"Punta Cana, Dominican Republic {\\&} Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2021.emnlp-tutorials.4\",\n    pages = \"17--21\",\n}",
  "metadata": {
    "title": "Multi-domain Multilingual Question Answering",
    "description": "This post expands on the EMNLP 2021 tutorial on Multi-domain Multilingual Question Answering and highlights key insights and takeaways.",
    "keywords": null,
    "author": null,
    "og:site_name": "ruder.io",
    "og:type": "article",
    "og:title": "Multi-domain Multilingual Question Answering",
    "og:description": "This post expands on the EMNLP 2021 tutorial on Multi-domain Multilingual Question Answering and highlights key insights and takeaways.",
    "og:url": "https://www.ruder.io/multi-qa-tutorial/",
    "og:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2021/12/rc_olympics.png",
    "og:image:width": "1200",
    "og:image:height": "600",
    "twitter:card": "summary_large_image",
    "twitter:title": "Multi-domain Multilingual Question Answering",
    "twitter:description": "This post expands on the EMNLP 2021 tutorial on Multi-domain Multilingual Question Answering and highlights key insights and takeaways.",
    "twitter:url": "https://www.ruder.io/multi-qa-tutorial/",
    "twitter:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2021/12/rc_olympics.png",
    "twitter:label1": "Written by",
    "twitter:data1": "Sebastian Ruder",
    "twitter:label2": "Filed under",
    "twitter:site": "@seb_ruder",
    "article:published_time": "2021-12-06T12:38:00.000Z",
    "article:modified_time": "2021-12-07T22:19:39.000Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/e29f1e8231ed/code/body-000.txt",
      "chars": 518,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2021/12/rc_olympics.png",
      "bytes": 425978,
      "reused": true,
      "local_path": "attachments/e29f1e8231ed/images/001.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2021/12/rc_olympics.png",
      "alt": "Multi-domain Multilingual Question Answering",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/orqa-1.png",
      "bytes": 776601,
      "reused": true,
      "local_path": "attachments/e29f1e8231ed/images/005.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/orqa-1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/reading_comprehension-1.png",
      "bytes": 768320,
      "reused": true,
      "local_path": "attachments/e29f1e8231ed/images/006.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/reading_comprehension-1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/bert_rc.png",
      "bytes": 401715,
      "reused": true,
      "local_path": "attachments/e29f1e8231ed/images/007.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/bert_rc.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/domain_granularity.png",
      "bytes": 234201,
      "reused": true,
      "local_path": "attachments/e29f1e8231ed/images/008.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/domain_granularity.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/rc_olympics.png",
      "bytes": 380026,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-b93f5dd2e8d2099129b2.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/rc_olympics.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/qagen.png",
      "bytes": 516523,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-a0d0f1dea2cca7df899a.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/qagen.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/l2_regularization-1.png",
      "bytes": 700399,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-ebc4029f4fea0b85e5b9.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/l2_regularization-1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/mlqa_annotation.png",
      "bytes": 363059,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-578b59416a3908cf795a.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/mlqa_annotation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/xor_tydiqa.png",
      "bytes": 632808,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-093d248aa63215501c9c.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/xor_tydiqa.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/utility_nlp_applications.png",
      "bytes": 456716,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-e29562cb2de292932f53.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/utility_nlp_applications.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/multichecklist_evaluation.png",
      "bytes": 613125,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-e237dfb92e8ff5d6dbe3.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/multichecklist_evaluation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/multilingual_rc_evaluation.png",
      "bytes": 100759,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-fd0bb5b686de8c51c836.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/multilingual_rc_evaluation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/multilingual_rc_flowchart.png",
      "bytes": 152877,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-69a0f1778bc2a238c149.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/multilingual_rc_flowchart.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/open-retrieval_translate-test.png",
      "bytes": 237022,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-5f0b31bd8e6bd8361d3c.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/open-retrieval_translate-test.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/cross-lingual_data_expansion.png",
      "bytes": 421470,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-384c7c1abad1899de932.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/cross-lingual_data_expansion.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/mdpr_mgen.png",
      "bytes": 437688,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-6f351bf552e740f050b1.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/mdpr_mgen.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/situatedqa.png",
      "bytes": 223053,
      "reused": false,
      "local_path": "attachments/e29f1e8231ed/images/body-6cfa795f60002d2f5495.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2021/11/situatedqa.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.ruder.io/author/sebastian/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://tinyurl.com/multi-qa-tutorial?ref=ruder.io",
      "label": "EMNLP 2021 tutorial on Multi-domain Multilingual Question Answering",
      "type": "a"
    },
    {
      "href": "https://researcher.watson.ibm.com/researcher/view_person_pubs.php?person=us-avi&t=1&ref=ruder.io",
      "label": "Avi Sil",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/P17-1171/?ref=ruder.io",
      "label": "Chen et al., 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1901.08634?ref=ruder.io",
      "label": "Alberti et al., 2019",
      "type": "a"
    },
    {
      "href": "https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf?ref=ruder.io",
      "label": "Robertson et al., 2009",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf?ref=ruder.io",
      "label": "tf-idf",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.emnlp-main.550/?ref=ruder.io",
      "label": "Karpukhin et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1608.07836?ref=ruder.io",
      "label": "Plank et al., 2016",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.acl-main.740/?ref=ruder.io",
      "label": "Gururangan et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/D16-1264/?ref=ruder.io",
      "label": "Rajpurkar et al., 2016",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/Q19-1026/?ref=ruder.io",
      "label": "Kwiatkowski et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1903.00161?ref=ruder.io",
      "label": "Dua et al., 2019",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/P16-1145/?ref=ruder.io",
      "label": "Hewlett et al., 2016",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/Q18-1023/?ref=ruder.io",
      "label": "Kočiský et al., 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1511.02301?ref=ruder.io",
      "label": "Hill et al., 2016",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1610.00956?ref=ruder.io",
      "label": "Bajgar et al., 2016",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/D13-1020/?ref=ruder.io",
      "label": "Richardson et al., 2013",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/L18-1564?ref=ruder.io",
      "label": "Modi et al., 2016",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/N16-1098/?ref=ruder.io",
      "label": "Mostafazedh et al., 2016",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1803.05457?ref=ruder.io",
      "label": "Clark et al., 2018",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=HJgJtT4tvB&ref=ruder.io",
      "label": "Yu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/D17-1082/?ref=ruder.io",
      "label": "Lai et al., 2017",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/W17-2623/?ref=ruder.io",
      "label": "Trischler et al., 2017",
      "type": "a"
    },
    {
      "href": "https://proceedings.neurips.cc/paper/2015/file/afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf?ref=ruder.io",
      "label": "Hermann et al., 2015",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.eacl-main.106/?ref=ruder.io",
      "label": "Soleimani et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.acl-main.117/?ref=ruder.io",
      "label": "Castelli et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/P15-2114/?ref=ruder.io",
      "label": "dos Santos et al., 2015",
      "type": "a"
    },
    {
      "href": "https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-015-0564-6?ref=ruder.io",
      "label": "Tsatsaronis et al., 2015",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.naacl-main.365/?ref=ruder.io",
      "label": "Dasigi et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.nlpcovid19-acl.18/?ref=ruder.io",
      "label": "Möller et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2004.11339?ref=ruder.io",
      "label": "Tang et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.nlpcovid19-2.1/?ref=ruder.io",
      "label": "Lee et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/Q19-1016/?ref=ruder.io",
      "label": "Reddy et al., 2019",
      "type": "a"
    },
    {
      "href": "https://ojs.aaai.org//index.php/AAAI/article/view/6398?ref=ruder.io",
      "label": "Rogers et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.adaptnlp-1.10/?ref=ruder.io",
      "label": "Guo et al., 2021",
      "type": "a"
    },
    {
      "href": "https://leaderboard.allenai.org/orb/submissions/about?ref=ruder.io",
      "label": "ORB",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1912.12598?ref=ruder.io",
      "label": "Dua et al., 2019",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/D19-1254/?ref=ruder.io",
      "label": "Wang et al., 2019",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.emnlp-main.439/?ref=ruder.io",
      "label": "Shakeri et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.emnlp-main.754/?ref=ruder.io",
      "label": "Yue et al., 2021",
      "type": "a"
    },
    {
      "href": "https://ojs.aaai.org/index.php/AAAI/article/view/6245?ref=ruder.io",
      "label": "Cao et al., 2020",
      "type": "a"
    },
    {
      "href": "https://ruder.io/recent-advances-lm-fine-tuning/?ref=ruder.io",
      "label": "domain-adaptive fine-tuning",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P18-1031/?ref=ruder.io",
      "label": "Howard & Ruder, 2018",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.acl-main.740/?ref=ruder.io",
      "label": "Gururangan et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1901.08746?ref=ruder.io",
      "label": "Lee et al., 2019",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/D19-1371/?ref=ruder.io",
      "label": "Beltagy et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1911.00202?ref=ruder.io",
      "label": "Xu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.emnlp-main.440/?ref=ruder.io",
      "label": "Zhang et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.semanticscholar.org/paper/Synthetic-Target-Domain-Supervision-for-Open-QA-Reddy-Iyer/824666e5e6721bc83a1768579062077d719f7089?ref=ruder.io",
      "label": "Reddy et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/P19-1485/?ref=ruder.io",
      "label": "Talmor & Berant, 2019",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.findings-emnlp.171/?ref=ruder.io",
      "label": "Khashabi et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.nist.gov/publications/clir-evaluation-trec?ref=ruder.io",
      "label": "CLIR at TREC",
      "type": "a"
    },
    {
      "href": "https://staff.fnwi.uva.nl/m.derijke/Publications/Files/clef_qa_03_overview_proceedings.pdf?ref=ruder.io",
      "label": "Magnini et al., 2003",
      "type": "a"
    },
    {
      "href": "http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.495.4731&rep=rep1&type=pdf&ref=ruder.io",
      "label": "Gillard et al., 2006",
      "type": "a"
    },
    {
      "href": "https://www.isical.ac.in/~fire/data/working-notes_2012/adhoc/adhoc_ISM.pdf?ref=ruder.io",
      "label": "Yadav et al., 2012",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/C18-1038/?ref=ruder.io",
      "label": "Zhao & Zhao, 2018",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.acl-main.330/?ref=ruder.io",
      "label": "Xu et al., 2020",
      "type": "a"
    },
    {
      "href": "https://www.researchgate.net/profile/Maciej-Piasecki/publication/272685856_Open_dataset_for_development_of_Polish_Question_Answering_systems/links/560f8ff708aec422d1133caa/Open-dataset-for-development-of-Polish-Question-Answering-systems.pdf?ref=ruder.io",
      "label": "Marcinczuk et al., 2013",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/2020.acl-main.653/?ref=ruder.io",
      "label": "Lewis et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2103.12028?ref=ruder.io",
      "label": "Caswell et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.acl-main.421/?ref=ruder.io",
      "label": "Artetxe et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.emnlp-main.477/?ref=ruder.io",
      "label": "Roy et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/P19-1227/?ref=ruder.io",
      "label": "Liu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Cloze_test?ref=ruder.io",
      "label": "Cloze format",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.tacl-1.30/?ref=ruder.io",
      "label": "Clark et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.naacl-main.46/?ref=ruder.io",
      "label": "Asai et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.mrl-1.12/?ref=ruder.io",
      "label": "Zhang et al., 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2110.07150?ref=ruder.io",
      "label": "Muller et al., 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2007.15207?ref=ruder.io",
      "label": "Longpre et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.emnlp-main.185/?ref=ruder.io",
      "label": "Ponti et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.acl-long.102/?ref=ruder.io",
      "label": "Lin et al., 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2110.06733?ref=ruder.io",
      "label": "Blasi et al., 2021",
      "type": "a"
    },
    {
      "href": "http://cs.haifa.ac.il/~shuly/publications/vered.pdf?ref=ruder.io",
      "label": "Volanksy et al., 2015",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2103.16613?ref=ruder.io",
      "label": "Valentim et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.eacl-main.86.pdf?ref=ruder.io",
      "label": "Lewis et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.emnlp-main.618/?ref=ruder.io",
      "label": "Artetxe et al., 2020",
      "type": "a"
    },
    {
      "href": "https://people.ict.usc.edu/~gordon/publications/AAAI-SPRING11A.PDF?ref=ruder.io",
      "label": "Roemmele et al., 2011",
      "type": "a"
    },
    {
      "href": "https://academic.oup.com/applij/article-abstract/4/2/91/167524?redirectedFrom=fulltext&ref=ruder.io",
      "label": "Thomas, 1983",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.acl-long.118/?ref=ruder.io",
      "label": "Asai & Choi, 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2107.12708?ref=ruder.io",
      "label": "Rogers et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.acl-main.442/?ref=ruder.io",
      "label": "Ribeiro et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.emnlp-main.802/?ref=ruder.io",
      "label": "Ruder et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.emnlp-main.479.pdf?ref=ruder.io",
      "label": "Jiang et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.eacl-main.284.pdf?ref=ruder.io",
      "label": "Kassner et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.findings-emnlp.195/?ref=ruder.io",
      "label": "∀ et al., 2020",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=SkeHuCVFDr&ref=ruder.io",
      "label": "Zhang et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.mrqa-1.15/?ref=ruder.io",
      "label": "Risch et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/N19-1423/?ref=ruder.io",
      "label": "Devlin et al., 2019",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.acl-main.747/?ref=ruder.io",
      "label": "Conneau et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.naacl-main.41/?ref=ruder.io",
      "label": "Xue et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.emnlp-main.617/?ref=ruder.io",
      "label": "Pfeiffer et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2003.11080?ref=ruder.io",
      "label": "Hu et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.emnlp-main.363/?ref=ruder.io",
      "label": "Lauscher et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2110.04374?ref=ruder.io",
      "label": "Kirstain et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.acl-short.79.pdf?ref=ruder.io",
      "label": "Debnath et al., 2021",
      "type": "a"
    },
    {
      "href": "http://doras.dcu.ie/16441/1/Efficient_Question_Answering_with_Question_Decomposition_and_Multiple_Answer_Streams.pdf?ref=ruder.io",
      "label": "Hartrumpf et al., 2008",
      "type": "a"
    },
    {
      "href": "http://research.nii.ac.jp/~ntcadm/workshop/OnlineProceedings8/NTCIR/06-NTCIR8-CCLQA-LinC.pdf?ref=ruder.io",
      "label": "Lin & Kuo, 2010",
      "type": "a"
    },
    {
      "href": "http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings8/NTCIR/04-NTCIR8-CCLQA-ShimaH.pdf?ref=ruder.io",
      "label": "Shima & Mitamura, 2010",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1809.03275?ref=ruder.io",
      "label": "Asai et al., 2018",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/D19-1607/?ref=ruder.io",
      "label": "Hsu et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2108.08787?ref=ruder.io",
      "label": "Zhang et al., 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2107.11976?ref=ruder.io",
      "label": "Asai et al., 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2109.01628?ref=ruder.io",
      "label": "Shi et al., 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1911.04118?ref=ruder.io",
      "label": "Garg et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.findings-emnlp.281/?ref=ruder.io",
      "label": "Faisal et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.emnlp-main.586/?ref=ruder.io",
      "label": "Zhang & Choi, 2021",
      "type": "a"
    },
    {
      "href": "http://www2015.thewebconf.org/documents/proceedings/companion/p853.pdf?ref=ruder.io",
      "label": "Raghavi et al., 2015",
      "type": "a"
    },
    {
      "href": "http://personales.upv.es/prosso/resources/BanerjeeEtAl_MultiLingMine16.pdf?ref=ruder.io",
      "label": "Banerjee et al., 2016",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/W18-3204/?ref=ruder.io",
      "label": "Chandu et al., 2018",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/W18-3205/?ref=ruder.io",
      "label": "Gupta et al., 2018",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.acl-long.131.pdf?ref=ruder.io",
      "label": "Doğruöz et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.naacl-main.393/?ref=ruder.io",
      "label": "Krishna et al., 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1911.10470?ref=ruder.io",
      "label": "Asai et al., 2020",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/D18-1241/?ref=ruder.io",
      "label": "Choi et al., 2018",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.coling-main.603/?ref=ruder.io",
      "label": "Ramponi & Plank, 2020",
      "type": "a"
    },
    {
      "href": "https://github.com/danqi/acl2020-openqa-tutorial?ref=ruder.io",
      "label": "ACL 2020 tutorial",
      "type": "a"
    },
    {
      "href": "https://tinyurl.com/xlingual?ref=ruder.io",
      "label": "ACL 2019 tutorial",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.emnlp-tutorials.4/?ref=ruder.io",
      "label": "the tutorial",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:02Z",
  "source_url": "https://www.ruder.io/multi-qa-tutorial/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1901.08634",
      "bytes": 180580,
      "reused": false,
      "local_path": "papers/repair/e83559720bad0f947c1d.pdf"
    },
    {
      "status": "ok",
      "url": "https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf?ref=ruder.io",
      "bytes": 423997,
      "reused": false,
      "local_path": "papers/repair/3273fd18be40087c7c09.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1608.07836",
      "bytes": 273244,
      "reused": false,
      "local_path": "papers/repair/15b3b816d50c2902b62c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1903.00161",
      "bytes": 1989064,
      "reused": false,
      "local_path": "papers/repair/13d2364f05d75a824d1e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1511.02301",
      "bytes": 1811140,
      "reused": false,
      "local_path": "papers/repair/0269f6b60742e2f6df8b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1610.00956",
      "bytes": 468143,
      "reused": false,
      "local_path": "papers/repair/9db6d1df712584e4419c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1803.05457",
      "bytes": 290564,
      "reused": false,
      "local_path": "papers/repair/87cdebbac7c0abeabd35.pdf"
    },
    {
      "status": "ok",
      "url": "https://proceedings.neurips.cc/paper/2015/file/afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf?ref=ruder.io",
      "bytes": 660392,
      "reused": false,
      "local_path": "papers/repair/fc8506599c5efa02659f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2004.11339",
      "bytes": 120805,
      "reused": false,
      "local_path": "papers/repair/51aba510ac9fa46a8cf6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1912.12598",
      "bytes": 107360,
      "reused": false,
      "local_path": "papers/repair/a55c008e3cc2cc42370b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1901.08746",
      "bytes": 1403854,
      "reused": false,
      "local_path": "papers/repair/fde6e03d4f14096f262c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1911.00202",
      "bytes": 406422,
      "reused": false,
      "local_path": "papers/repair/1f711f725a6517393dd0.pdf"
    },
    {
      "status": "ok",
      "url": "https://staff.fnwi.uva.nl/m.derijke/Publications/Files/clef_qa_03_overview_proceedings.pdf?ref=ruder.io",
      "bytes": 202208,
      "reused": false,
      "local_path": "papers/repair/27e68ba3eb05741d40c8.pdf"
    },
    {
      "status": "ok",
      "url": "https://www.isical.ac.in/~fire/data/working-notes_2012/adhoc/adhoc_ISM.pdf?ref=ruder.io",
      "bytes": 98650,
      "reused": false,
      "local_path": "papers/repair/663b762c494ee98de797.pdf"
    },
    {
      "status": "http_403",
      "url": "https://www.researchgate.net/profile/Maciej-Piasecki/publication/272685856_Open_dataset_for_development_of_Polish_Question_Answering_systems/links/560f8ff708aec422d1133caa/Open-dataset-for-development-of-Polish-Question-Answering-systems.pdf?ref=ruder.io",
      "retried_at": "2026-10-20T18:05:35Z"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2103.12028",
      "bytes": 460702,
      "reused": false,
      "local_path": "papers/repair/9e0a7e34c64b92799537.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2110.07150",
      "bytes": 523468,
      "reused": false,
      "local_path": "papers/repair/9ea205922fedbe81ba54.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2007.15207",
      "bytes": 1351038,
      "reused": true,
      "local_path": "papers/repair/e267de7fa087c8c40cd1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2110.06733",
      "bytes": 3215703,
      "reused": false,
      "local_path": "papers/repair/5003adb4762e8a493baa.pdf"
    },
    {
      "status": "invalid_file_type",
      "url": "http://cs.haifa.ac.il/~shuly/publications/vered.pdf?ref=ruder.io",
      "content_type": "text/html; charset=utf-8",
      "retried_at": "2026-10-20T18:05:37Z"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2103.16613",
      "bytes": 3300462,
      "reused": false,
      "local_path": "papers/repair/70ef167f61b46add37ec.pdf"
    },
    {
      "status": "ok",
      "url": "https://aclanthology.org/2021.eacl-main.86.pdf?ref=ruder.io",
      "bytes": 229598,
      "reused": false,
      "local_path": "papers/repair/143b265addbba5787883.pdf"
    },
    {
      "status": "http_404",
      "url": "https://people.ict.usc.edu/~gordon/publications/AAAI-SPRING11A.PDF?ref=ruder.io",
      "retried_at": "2026-10-20T18:05:36Z"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2107.12708",
      "bytes": 1177401,
      "reused": false,
      "local_path": "papers/repair/160d4893be939d7daf7e.pdf"
    },
    {
      "status": "ok",
      "url": "https://aclanthology.org/2020.emnlp-main.479.pdf?ref=ruder.io",
      "bytes": 687239,
      "reused": false,
      "local_path": "papers/repair/ce3771df39eb262d708e.pdf"
    },
    {
      "status": "ok",
      "url": "https://aclanthology.org/2021.eacl-main.284.pdf?ref=ruder.io",
      "bytes": 2070417,
      "reused": false,
      "local_path": "papers/repair/30bd4f74f3f23d051fe9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2003.11080",
      "bytes": 915662,
      "reused": true,
      "local_path": "papers/repair/38b7f5f2a070f3d837f3.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2110.04374",
      "bytes": 923154,
      "reused": false,
      "local_path": "papers/repair/badad8dcc386283e6595.pdf"
    },
    {
      "status": "ok",
      "url": "https://aclanthology.org/2021.acl-short.79.pdf?ref=ruder.io",
      "bytes": 260436,
      "reused": false,
      "local_path": "papers/repair/db3964cd90cc6907d455.pdf"
    },
    {
      "status": "ok",
      "url": "http://doras.dcu.ie/16441/1/Efficient_Question_Answering_with_Question_Decomposition_and_Multiple_Answer_Streams.pdf?ref=ruder.io",
      "bytes": 114423,
      "reused": false,
      "local_path": "papers/repair/828d74659a7ebf1ca3e0.pdf"
    },
    {
      "status": "ok",
      "url": "http://research.nii.ac.jp/~ntcadm/workshop/OnlineProceedings8/NTCIR/06-NTCIR8-CCLQA-LinC.pdf?ref=ruder.io",
      "bytes": 246686,
      "reused": false,
      "local_path": "papers/repair/2f8d8d4ec58281ba6ec7.pdf"
    },
    {
      "status": "ok",
      "url": "http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings8/NTCIR/04-NTCIR8-CCLQA-ShimaH.pdf?ref=ruder.io",
      "bytes": 211817,
      "reused": false,
      "local_path": "papers/repair/036ea0133a3ac1ac15d5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1809.03275",
      "bytes": 1187079,
      "reused": false,
      "local_path": "papers/repair/50c12c58bd2d41c654c8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2108.08787",
      "bytes": 870899,
      "reused": false,
      "local_path": "papers/repair/0069fe90035cb1adf757.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2107.11976",
      "bytes": 3569093,
      "reused": false,
      "local_path": "papers/repair/a9089e838cbc55d8f1ba.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2109.01628",
      "bytes": 196487,
      "reused": false,
      "local_path": "papers/repair/f7b95c3d6e56e82650aa.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1911.04118",
      "bytes": 2090120,
      "reused": true,
      "local_path": "papers/repair/db2b28e9c161d58c0f84.pdf"
    },
    {
      "status": "download_error",
      "url": "http://www2015.thewebconf.org/documents/proceedings/companion/p853.pdf?ref=ruder.io",
      "reason": "ConnectionError: HTTPConnectionPool(host='www2015.thewebconf.org', port=80): Max retries exceeded with url: /documents/proceedings/companion/p853.pdf?ref=ruder.io (Caused by NameResolutionError(\"HT",
      "retried_at": "2026-10-20T22:24:46Z"
    },
    {
      "status": "ok",
      "url": "http://personales.upv.es/prosso/resources/BanerjeeEtAl_MultiLingMine16.pdf?ref=ruder.io",
      "bytes": 280988,
      "reused": false,
      "local_path": "papers/repair/2489a073996393a9ba22.pdf"
    },
    {
      "status": "ok",
      "url": "https://aclanthology.org/2021.acl-long.131.pdf?ref=ruder.io",
      "bytes": 358721,
      "reused": false,
      "local_path": "papers/repair/d8c7641b61ab19ae36ec.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1911.10470",
      "bytes": 2803789,
      "reused": false,
      "local_path": "papers/repair/a9d485709909311d707e.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 34102,
  "expected_body_images": 18,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "download_error；http_403；http_404；invalid_file_type",
  "raw_html": "attachments/e29f1e8231ed/pages/source.html",
  "local_html": "attachments/e29f1e8231ed/pages/article.html",
  "body_base_path": "attachments/e29f1e8231ed/pages/",
  "body_markdown_path": "attachments/e29f1e8231ed/pages/article.md"
}