{ "source_file": "0930", "source_platform": "个人博客", "author": "Sebastian Ruder", "title_listed": "Unsupervised Cross-lingual Representation Learning", "keywords": "跨语言,无监督,表示学习", "domain": "文本处理", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://www.ruder.io/unsupervised-cross-lingual-learning/", "publish_date": "2019-10", "corpus_stage": "预训练/SFT", "notes": "无监督跨语言表示学习综述", "id": "fa35ac2d4b0c", "collected_at": "2026-10-02T14:16:07", "title": "Unsupervised Cross-lingual Representation Learning", "body_markdown": "cross-lingual\nUnsupervised Cross-lingual Representation Learning\nThis post expands on the ACL 2019 tutorial on Unsupervised Cross-lingual Representation Learning. It highlights key insights and takeaways and provides updates based on recent work, particularly unsupervised deep multilingual models.\nSebastian Ruder\n26 Oct 2019\n•\n20 min read\nShare\nThis post expands on the\nACL 2019 tutorial on Unsupervised Cross-lingual Representation Learning\n.\nThe tutorial was organised by\nIvan Vulić\n,\nAnders Søgaard\n, and me. In this post, I highlight key insights and takeaways and provide additional context and updates based on recent work. In particular, I cover\nunsupervised deep multilingual models\nsuch as multilingual BERT. You can see the structure of this post below:\nThe slides of the tutorial are\navailable online\n.\nIntroduction\nCross-lingual representation learning can be seen as an instance of transfer learning, similar to domain adaptation. The domains in this case are different languages.\nCross-lingual learning in the transfer learning taxonomy (\nRuder, 2019\n)\nMethods from domain adaptation have also been applied to cross-lingual transfer (\nPrettenhofer & Stein, 2011\n,\nWan et al., 2011\n). For a clearer distinction between domain adaptation and cross-lingual learning, have a look at\nthis section\n.\nViewing cross-lingual learning as a form of transfer learning can help us understand why it might be useful and when it might fail:\nTransfer learning is useful whenever the scenarios between which we transfer share some underlying structure. In a similar vein, languages share commonalities on many levels—from\nloanwords\nand\ncognates\non the lexical level, to the\nstructure of sentences\non the syntactic level, to the\nmeaning of words\non the semantic level. Learning about the structure of one language may therefore help us do better when learning a second language.\nTransfer learning struggles when the source and target settings are too different. Similarly, it is harder to transfer between languages that are dissimilar (e.g. that have different typological features in the\nWorld Atlas of Language Structures\n).\nSimilar to the\nbroader transfer learning landscape\n, much of the work on cross-lingual learning over the last years has focused on learning representations of words. While there have been approaches that learn sentence-level representations, only recently are we witnessing the emergence of\ndeep multilingual models\n. Have a look at\nthis survey\nfor an overview of the history of cross-lingual models. For more on transfer learning, check out\nthis recent post\n. Cross-lingual learning might be useful—but why should we care about applying NLP to other languages in the first place?\nThe Digital Language Divide\nThe language you speak shapes your experience of the world. This is known as the\nSapir-Whorf hypothesis\nand is contested. On the other hand, it is factual that the language you speak shapes your experience of the world\nonline\n. What language you speak determines your access to information, education, and even human connections. In other words, even though we think of the Internet as being open to anyone, there is a\ndigital language divide\nbetween dominant languages (mostly from the western world) and others.\nAs NLP technologies are becoming more commonplace, this divide extends to the technological level: At best, this means that non-English language speakers do not benefit from the latest features of digital assistants; at worst, algorithms are biased against or discriminate against speakers of non-English languages—we can see\noccurrences\nof this\nalready\ntoday\n. To ensure that non-English language speakers are not left behind and at the same time to offset the existing imbalance, we need to apply our models to non-English languages.\nThe NLP Resource Hierarchy\nIn current machine learning, the amount of available training data is the main factor that influences an algorithm's performance. Data that is useful for current NLP models can range from unlabelled data in the form of documents online or articles on Wikipedia, labelled data in the form of large curated datasets, or parallel corpora used in machine translation.\nWe can roughly differentiate between languages with many and languages with few data resources in a hierarchy, which can be seen below.\nA conceptual view of the NLP resource hierarchy\nThis approximately corresponds with a language's presence online. Note that many languages cannot be assigned clearly to a single level of the hierarchy. Some languages with few speakers such as\nAragonese\nor\nWaray-Waray\nhave a\ndisproportionally\nlarge Wikipedia\n, while for others such as\nLatvian\nparallel data\nis available due to their status as an official language. On the other hand, many languages with tens of millions of speakers such as\nXhosa\nand\nZulu\nhave barely more than\n1,000 Wikipedia articles\n.\nNevertheless, the existence of such a hierarchy highlights why unsupervised cross-lingual representation learning is promising: for most of the world's languages, no labelled data is available and creating labelled data is expensive as native speakers are under-represented online. In order to train models for such languages, we need to make use of unlabelled data and transfer learning from high-resource languages.\nCross-lingual Representations\nWhereas in the past it was necessary to learn language-specific models for every language and preprocessing and features needed to be applied to every language in isolation, cross-lingual representation learning enables us to apply the\ntransfer learning formula\nof pretraining and adaptation\nacross languages\n.\nThe transfer learning formula (\nNAACL 2019 Transfer learning tutorial\n)\nIn this context, we will talk about a (high-resource) source language \\(L_1\\) where unlabelled data and labelled data for a particular task are available, and a (low-resource) target language \\(L_2\\) where only unlabelled data is available. For cross-lingual transfer learning, we add one additional step to the transfer learning recipe above:\nPretraining\n: Learn\ncross-lingual\nrepresentations that are shared across languages. This is what the rest of this post is about.\nAdaptation\n: Adapt the model to the labelled data of the desired task in \\(L_1\\) by learning task-specific parameters on top of the cross-lingual representations. The cross-lingual parameters are often kept fixed (this is the same as\nfeature extraction\n). As the task-specific parameters are learned on top of cross-lingual ones, they are expected to transfer to the other language.\nZero-shot transfer\n: The model, which captures cross-lingual information, can be applied directly to perform inference on data of the target language.\nIf labelled data in the target language is also available, then the model can be fine-tuned jointly on both languages in Step 2.\nThroughout this post, I will focus on approaches that learn cross-lingual representations as the means for transferring across languages. Depending on the task, there are other ways of transferring information across languages such as by domain adaptation (as seen above), annotation projection (\nPadó & Lapata, 2009\n;\nNi et al., 2017\n;\nNicolai & Yarowsky, 2019\n), distant supervision (\nPlank & Agić, 2018\n) or machine translation (MT;\nZhou et al., 2016\n;\nEger et al., 2018\n).\nWhy not Machine Translation?\nIn light of the recent success of\nmassively multilingual\nand unsupervised machine translation (\nLample et al., 2018\n;\nArtetxe et al., 2019\n), an obvious question is why we would not just use MT to translate the training data in our source language to the target language and train a model on the translated data—or translate the test set and apply our source model directly to it, which often works better in practice. If an MT system is available, this can be a strong baseline (\nConneau et al., 2018\n) and unsupervised MT in particular can be useful for low-resource languages (\nLample et al., 2018\n).\nHowever, an MT system for the desired language pair may not be easily available and can be expensive to train. In addition, models trained on machine-translated data often underperform\nstate-of-the-art deep multilingual models\n. MT struggles with distant language pairs and domain mismatches (\nGuzmán et al., 2019\n). It is also not a fit for all tasks; the performance of translation-based models for question answering tasks depends heavily on the translation quality of named entities (\nLiu et al., 2019\n). For sequence labelling tasks, MT requires projecting annotations across languages, itself a difficult problem (\nAkbik & Vollgraf, 2018\n).\nIn general, however, MT and the methods described in this post are not at odds but complementary. Cross-lingual word representations are used to initialise unsupervised MT (\nArtetxe et al., 2019\n) and cross-lingual representations have been shown to improve neural MT performance (\nLample & Conneau, 2019\n). Techniques from MT such as word alignment have also inspired much work in cross-lingual representation learning (\nRuder et al., 2019\n). Finally, a multilingual model benefits from being fine-tuned on translations of the labelled data in multiple languages (\nConneau et al., 2019\n).\nMain Framework\nWe now discuss how to learn unsupervised representations, starting at the word level. The main framework for learning cross-lingual word embeddings (CLWEs) consists of\nmapping-based\nor\npost-hoc alignment\napproaches. For an overview of other approaches, have a look at\nour survey\nor\nbook\non the topic.\nMapping-based methods are currently the preferred approach due to their ease and efficiency of use and conceptual simplicity. They only require monolingual word embeddings trained in each language and then simply learn a linear projection that maps between the embedding spaces. The general framework can be seen in the Figure below.\nA general framework for projection-based cross-lingual word embeddings (\nGlavaš et al., 2019\n)\nGiven monolingual embedding spaces \\(\\mathbf{X}_{L_1} = \\{\\mathbf{x}^i_{L_1} \\}^{|V_{L_1}|}_{i=1} \\) and \\(\\mathbf{X}_{L_2} = \\{\\mathbf{x}^j_{L_2} \\}^{|V_{L_2}|}_{j=1} \\) in two languages where \\(\\mathbf{x}\\) is a word embedding and \\(|V|\\) is the size of the vocabulary, we perform the following steps (\nGlavaš et al., 2019\n):\nConstruct a seed translation dictionary\n. The dictionary \\(D = \\{w^i_{L_1}, w^j_{L_2} \\}^K\\) contains \\(K\\) pairs of words and their translations.\nCreate word-aligned monolingual subspaces\n. We create subspaces \\(\\mathbf{X}_S = \\{\\mathbf{x}\ni_{L_1}\\}\nK_{i=1} \\) and \\(\\mathbf{X}_T = \\{\\mathbf{x}\nj_{L_2}\\}\nK_{j=1} \\) where \\(\\mathbf{X}_S \\in \\mathbb{R}^{K \\times d}\\) and \\(\\mathbf{X}_T \\in \\mathbb{R}^{K \\times d}\\) where \\(d\\) is the dimensionality of the word embeddings. We do this simply by looking up the vectors of the words and their translations in the monolingual embedding spaces, \\( \\{w^i_{L_1} \\} \\) from \\(\\mathbf{X}_{L_1}\\) and \\( \\{w^j_{L_2} \\} \\) from \\(\\mathbf{X}_{L_2}\\) respectively.\nLearn a mapping between the subspaces to a common space\n. Specifically, we project the matrices \\(\\mathbf{X}_S\\) and \\(\\mathbf{X}_T\\) into a common space \\(\\mathbf{X}_{CL}\\). In the general setting, we learn two projection matrices \\(\\mathbf{W}_{L_1} \\in \\mathbb{R}^{d \\times d} \\) and \\(\\mathbf{W}_{L_2} \\in \\mathbb{R}^{d \\times d}\\) to project \\(\\mathbf{X}_{L_1}\\) and \\(\\mathbf{X}_{L_2}\\) respectively to the shared space.\nWe often treat one of the monolingual spaces \\(\\mathbf{X}_{L_1}\\) (typically the English space) as the cross-lingual space \\(\\mathbf{X}_{CL}\\) and only learn a mapping \\(\\mathbf{W}_{L_2}\\) from \\(\\mathbf{X}_{L_2}\\) to this space. The standard approach for learning this mapping is to minimise the Euclidean distance between the representations of words \\( \\mathbf{X}_T \\) and their projected translations \\( \\mathbf{X}_S\\mathbf{W} \\) in the dictionary \\(D\\) (\nMikolov et al., 2013\n):\n\\(\\mathbf{W}_{L_2} = \\arg \\min_{\\mathbf{W}} \\| \\mathbf{X}_S\\mathbf{W} - \\mathbf{X}_T \\|_2 \\)\nAfter having learned this mapping, we can now project a word embedding \\(\\mathbf{x}_{L_2}\\) from \\(\\mathbf{X}_{L_2}\\) simply as \\(\\mathbf{W}_{L_2} \\mathbf{x}_{L_2} \\) to the space of \\(\\ \\mathbf{X}_{L_1}\\). We can see how this mapping looks like geometrically below.\nVisualisation of the mapping-based approach between Swahili and English subspaces (inspired by slides from Eneko Agirre & Mikel Artetxe)\nMore complex mappings (such as with a deep neural network) have been observed to lead to poorer performance (\nMikolov et al., 2013\n). Better word translation results can be achieved when \\(\\mathbf{W}_{L_2}\\) is constrained to be orthogonal (\nXing et al., 2015\n,\nArtetxe et al., 2016\n). Intuitively, this ensures that when \\(\\mathbf{W}_{L_2}\\) vectors are projected to the \\(\\mathbf{X}_{CL}\\) space that the structure of the monolingual embedding space is preserved. If \\(\\mathbf{W}_{L_2}\\) is orthogonal, then the optimisation problem is known as the so-called Procrustes problem (\nSchönemann, 1966\n), which has a closed form solution:\n\\(\n\\begin{align}\n\\begin{split}\n\\mathbf{W}_{L_2} & = \\mathbf{UV}^\\top, \\text{with} \\\\\n\\mathbf{U \\Sigma V}^\\top & = SVD(\\mathbf{X}_T \\mathbf{X}_S{}^\\top)\n\\end{split}\n\\end{align}\n\\)\nAlmost all projection-based methods, whether they are supervised or unsupervised, solve the Procrustes problem to learn a linear mapping in Step 3. The main part where they differ is in how they obtain the initial seed dictionary in Step 1. Recent supervised approaches (\nArtetxe et al., 2017\n) have been shown to achieve reasonable performance using dictionaries of only 25-40 translation pairs. The main idea that enables learning from such limited supervision is bootstrapping or self-learning.\nSelf-learning\nStarting from a seed dictionary, recent approaches employ a\nself-learning procedure\n(\nArtetxe et al., 2017\n), which can be seen below.\nThe self-learning loop (Credit: Eneko Agirre)\nWe learn an initial mapping based on the seed dictionary and use this mapping to project \\(\\mathbf{X}_{L_2}\\) vectors to the target space \\(\\mathbf{X}_{CL}\\). In this cross-lingual space, \\(L_1\\) words should be close to their translations in \\(L_2\\). We can now use the \\(L_1\\) words and their nearest neighbours in \\(L_2\\) to build our new dictionary. In practice, people often only consider frequent words and mutual nearest neighbours, i.e. words that are nearest neighbours from both \\(L_1\\) and \\(L_2\\) directions (\nVulić et al., 2016\n;\nLample et al., 2018\n).\nThe framework that is typically used in practice consists of additional components (\nArtetxe et al., 2018\n) that can be seen in the Figure below.\nThe general unsupervised cross-lingual word embedding framework (\nVulić et al., 2019\n)\nIn particular, beyond the induction of the initial seed dictionary (C1), the learning of the projections, and the self-learning (C3), there are several pre- and post-processing steps that are often used in practice. You can have a look at\nthe slides\nor refer to\nArtetxe et al. (2018)\nfor an overview.\nUnsupervised Seed Induction\nUnsupervised seed induction aims to find a good initial dictionary that we can use to bootstrap the self-learning loop. We can differentiate approaches based on whether they learn an initial mapping by minimising the distance between the \\(\\mathbf{X}_{L_1}\\) or \\(\\mathbf{X}_{L_2}\\) spaces—often with an adversarial component—or whether they rely on a non-adversarial approach.\nAdversarial approaches\nAdversarial approaches are inspired by\ngenerative adversarial networks (GANs)\n. The generator is parameterised by our projection matrix \\(\\mathbf{W}_{L_2}\\) while the discriminator is a separate neural network that tries to discriminate between true embeddings from \\(\\mathbf{X}_{L_2}\\) and projected embeddings \\(\\mathbf{W}_{L_1} \\mathbf{X}_{L_1}\\) as can be seen below.\nThe adversarial approach for learning cross-lingual word embeddings\nIn order to fool the discriminator, the generator has to transform \\(\\mathbf{X}_{L_1}\\) in such a way that it matches the distribution of \\(\\mathbf{X}_{L_2}\\). The underlying hypothesis is that the transformation that makes the distributions as similar as possible also puts words close to their translations in the cross-lingual space \\(\\mathbf{X}_{CL}\\). GANs can be notoriously brittle. The first approach that worked more robustly in certain settings was proposed by\nConneau et al. (2018)\n.\nOther ways have been proposed to minimise the distance between the distributions of \\(\\mathbf{X}_{L_1}\\) and \\(\\mathbf{X}_{L_2}\\) that leverage Wasserstein GANs, Earth Mover's distance, and optimal transport. For more details about these approaches and their connections, have a look at\nthe corresponding slides\n.\nWeak supervision and second-order similarity\nAnother strategy for obtaining a seed lexicon is through the use of heuristics. If the writing systems of two languages share the same alphabet, then there will be many words that are spelled the same in both languages, many of them named entities such as \"New York\" and \"Barack Obama\". A list of pairs of such identically spelled words can be used as a weakly supervised seed dictionary (\nSøgaard et al., 2018\n). Even if languages do not share alphabets, many languages still use Arabic numerals, which can be used as a seed dictionary (\nArtetxe et al., 2017\n).\nSuch a weakly supervised seed dictionary is available for most language pairs and works surprisingly well, often outperforming purely unsupervised approaches. Intuitively, despite being noisy, a dictionary consisting of identically spelled words provides a sufficient number of anchor points for learning an initial mapping.\nStill, it would be useful to have a seed dictionary that is independent of a language's writing system. Rather than requiring that translations should be similar (e.g. spelled the same), we can go one step further and require that translations should be similar to other words across languages\nin the same way\n(\nArtetxe et al. 2018\n). In other words, the similarity distributions of translations within each language should be similar.\nSpecifically, if we calculate the cosine similarity of one word with all other words in the same language, sort the values, and normalise, we obtain the following distributions for \"two\", \"due\" and \"cane\":\nIntralingual similarity distributions of three words (\nArtetxe et al., 2018\n).\nThe intralingual similarity distributions of translations such as \"two\" and \"due\" should be more similar compared to other words such as \"cane\". Based on this insight, we can use the nearest neighbours of our similarity distributions across languages as the initial dictionary. This notion of second-order similarity (the similarity of similarity distributions) is a powerful concept and is also leveraged by distance metrics such as the\nGromov-Wasserstein distance\n.\nSystematic Comparisons\nUnsupervised cross-lingual word embeddings are theoretically interesting. They can teach us how to better leverage monolingual data across languages. We can also use them to learn more about how different languages relate to each other.\nHowever, supervision in the form of translation pairs is available for many languages and weak supervision is readily available. Unsupervised methods are thus only of practical interest if they are able to outperform to their supervised counterparts.\nInitial papers claimed that unsupervised methods are indeed competitive and even outperform the basic supervised Procrustes method. Recent studies, however, control for the\nadditional components in the framework\nand compare unsupervised and supervised methods on equal footing. The picture that has emerged is that current unsupervised methods generally underperform supervised methods with 500-1,000 seed translation pairs (\nGlavaš et al., 2019\n;\nVulić et al., 2019\n) as can be seen in the Figure below.\nComparison of BLI performances of a fully unsupervised method (UNSUPER), a supervised method (SUPER), and supervised methods with self-learning (+SL +NOD, +SL +SYM;\nVulić et al., 2019\n).\nMoreover, even the most robust unsupervised mapping-based approaches fail for many language pairs. This has to do with the topology of the monolingual embedding spaces: Unsupervised methods rely on the assumption that the embedding spaces are approximately isomorphic, i.e. that they have similar structure  (\nSøgaard et al., 2018\n). If this is not the case, then the unsupervised seed induction step fails.\nThe structure of embedding spaces can be different if embeddings belong to dissimilar languages but also if they were trained on different domains or using different algorithms (\nSøgaard et al., 2018\n;\nHartmann et al., 2018\n). Different methods such as a combination of several independent linear maps (\nNakashole, 2018)\n, iterative normalisation (\nZhang et al., 2019)\n, or incrementally adding new languages to the multilingual space (\nHeymann et al., 2019\n) have been proposed to align such non-isomorphic embedding spaces. Recent work has also shown that—controlling for additional components—the standard GAN approach is competitive with more advanced GANs such as Wasserstein GAN and with the second-order similarity initialisation (\nHartmann et al., 2019\n).\nUnsupervised Deep Models\nSupervised deep models that learn from parallel sentences or documents have been proposed before (see Sections 7 and 8 of\nthis survey\n). In light of the success of pretrained language models, similar techniques have recently been applied to train unsupervised deep cross-lingual representations.\nJoint models\nThe most prominent example in this line of work is\nmultilingual BERT\n(mBERT), a BERT-base model that was jointly trained on the corpora of 104 languages with a shared vocabulary of 110k subword tokens.\nmBERT is trained using masked language modelling (MLM)—with no explicit supervision—but has nevertheless been shown to learn cross-lingual representations that generalise surprisingly well to other languages via zero-shot transfer (\nPires et al., 2019\n;\nWu & Dredze, 2019\n). This generalisation ability has been attributed to three factors: 1) identical subwords in the shared vocabulary acting as anchor points for learning an alignment (similar to the weak supervision of CLWEs); 2) joint training across multiple languages that spreads this effect; and 3) deep cross-lingual representations that go beyond vocabulary memorisation and generalise across languages.\nRecent work (\nArtetxe et al., 2019\n;\nAnonymous et al., 2019\n;\nWu et al., 2019\n), however, shows that a shared vocabulary is not required for learning unsupervised deep cross-lingual representations.\nArtetxe et al. (2019)\nadditionally demonstrate that joint training is unnecessary and identify the vocabulary size per language as an important factor: Multilingual models with larger vocabulary sizes consistently perform better.\nExtensions to mBERT augment it with a supervised objective (\nLample and Conneau, 2019\n), which is inspired by\nbilingual skip-gram approaches\nand can be seen in the Figure below. Others add auxiliary pre-training tasks such as cross-lingual word recovery and paraphrase detection (\nHuang et al., 2019\n), encourage representations of translations to be similar (\nAnonymous, 2019\n), and scale it up (\nConneau et al., 2019\n). Joint training has been applied not only to Transformers but also to LSTM-based methods where initialisation with CLWEs has been found useful (\nMulcaire et al., 2019\n).\nThe supervised cross-lingual language modelling objective of\nLample and Conneau (2019\n)\nMapping-based approaches\nThe mapping-based approaches that we discussed previously have also been applied to the contextual representations of deep bilingual models. The main assumption behind these methods is that contextual representations—similar to their non-contextual counterparts—can also be aligned via a linear transformation. As can be seen in the Figure below, contextual representations cluster and have similar distributions across languages.\nPCA visualisation of contextual embeddings of the English word \"bear\" and its two possible Spanish translations, \"tener\" and \"oso\" in an aligned multilingual space (\nSchuster et al., 2019\n)\nFor contextual representations, the mapping can be done either on the token level or on the type level. On the token level, contextual representations of tokens in both languages can be aligned using word alignment information in parallel data (\nAldarmaki & Diab, 2019\n). On the type level, an aggregate representation of a word's contextual representations such as its mean can be used as its type embeddings (\nSchuster et al., 2019\n). Type embeddings across languages can then be aligned using the same mapping-based approaches as before.\nLiu et al. (2019)\nperform further analyses and find that type-level alignment performs best. Mapping-based methods have also recently been applied to BERT-based representations (\nAnonymous et al., 2019\n;\nWu et al., 2019\n).\nRather than learning a mapping between two existing pretrained models in \\(L_1\\) and \\(L_2\\), we can apply a model that was pretrained in a high-resource language \\(L_1\\) to a low-resource language \\(L_2\\) much more easily by simply learning token-level embeddings in \\(L_2\\) while keeping the model body fixed (\nArtetxe et al., 2019\n,\nAnonymous et al., 2019\n). The full process can be seen below. This alignment to a monolingual model may avoid the non-isomorphism of fixed monolingual vector spaces and the resulting model is competitive with jointly trained models.\nThe four steps of the monolingual transfer approach of\nArtetxe et al. (2019\n)\nFuture Directions\nBenchmarks\nA standard task for evaluating CLWEs is bilingual lexicon induction (BLI), which evaluates the quality of the cross-lingual embedding space by determining whether words are nearest neighbours of their translations in a test dictionary. Recently, several researchers have identified issues with the standard MUSE dictionaries (\nConneau et al., 2018\n) for this task, which only represent frequent words and are not morphologically diverse (\nCzarnowska et al., 2019\n;\nKementchedjhieva et al., 2019\n). BLI performance on words with lower frequency ranks drops off drastically as can be seen below.\nBLI performance vs frequency rank of words in MUSE and new dictionaries (\nCzarnowska et al., 2019\n)\nWe are ultimately interested in applying our methods to downstream tasks in low-resource languages. While BLI correlates well with certain downstream tasks for mapping-based approaches (\nGlavaš et al., 2019\n), the correlation is weaker for methods that are not based on an orthogonal transformation. Downstream evaluation is thus important to accurately gauge the performance of novel methods in the future.\nHowever, even existing cross-lingual downstream tasks have their problems. Cross-lingual document classification, a classic task for the evaluation of cross-lingual representations (\nKlementiev et al., 2012\n), mostly requires superficial keyword-matching and CLWEs outperform deep representations (\nArtetxe et al., 2019\n). A more recent multilingual benchmark, XNLI (\nConneau et al., 2018\n) may be plagued by similar artefacts as the MultiNLI dataset from which it has been derived (\nGururangan et al., 2018\n).\nOn the whole, we need more challenging benchmarks for evaluating multilingual models. Question answering might be a good candidate as it is a common probe for language understanding and has been shown to be harder to exploit. Recently, several cross-lingual question answering datasets have been proposed, including XQuAD (\nArtetxe et al., 2019\n), which extends SQuAD 1.1. to ten other languages, the open-domain XQA (\nLiu et al., 2019\n), and MLQA (\nLewis et al., 2019\n).\nUnderstanding representations\nBenchmarks are also important for better understanding the representations that our models learn. Along these lines, it is still unclear what makes monolingual embedding spaces non-isomorphic. Even embedding spaces between similar languages are not strictly isomorphic as can be seen below. Similar issues might affect the spaces of deep contextual representations. Identifying the cause of this phenomenon may help us gain a better understanding of our learning algorithms and the biases they encode in their representations. It may also shed light on the semantic similarities and differences between languages.\nNearest neighbour graphs of 10 most frequent nouns in English and their German translations (\nSøgaard et al., 2018\n)\nThere have been several studies analysing what CLWEs learn but it is still mostly unclear what representations learned by unsupervised deep multilingual models capture. As token-level alignment to a deep monolingual model achieves competitive results (\nArtetxe et al., 2019\n), deep multilingual models may still mostly learn lexical alignment. If this is the case, we need better methods that incentivise learning deep multilingual representations. Parallel data improves performance in practice (\nLample & Conneau, 2019\n), but for low-resource languages we would like to use as little supervision as possible.\nDeep representations from an English BERT model can be transferred to another language without any modification (besides learning new token embeddings;\nArtetxe et al., 2019\n). Consequently, it should be interesting to analyse what cross-lingual information monolingual models capture. In order to gain new insights, however, we need better benchmarks and evaluation protocols for the analysis of cross-lingual knowledge.\nPractical considerations\nIn the end, we'd like to learn cross-lingual representations in order to apply them to downstream tasks in low-resource languages. In light of this, we should not forget practical concerns of such low-resource scenarios. While zero-shot transfer is a focus of current methods,\nzero-shot transfer is not free\n. It requires access to training data from the same task and distribution in a high-resource language. We should thus continue investing in approaches that can be trained with few samples in a target language and that combine the best of both the monolingual and cross-lingual worlds, for instance via cross-lingual bootstrapping that can be seen below (\nEisenschlos et al., 2019\n).\nSteps of a cross-lingual bootstrapping method for zero-shot cross-lingual transfer (\nEisenschlos et al., 2019\n)\nIn addition, recent deep multilingual models are large and expensive to train. Training smaller multilingual models, for instance via distillation (\nTsai et al., 2019\n), will be key for deployment in resource-constrained scenarios.\nRecent wide-coverage parallel corpora such as JW-300 (\nAgić & Vulić, 2019\n) and WikiMatrix (\nSchwenk et al., 2019\n) enable the training of massively multilingual or supervised systems for many new low-resource languages where parallel data was previously not available. However, these resources still do not cover languages where only few unlabelled data is available. In order to apply our methods to tasks in such languages, we need to develop approaches that can learn representations from a limited number of samples. Resources that provide limited data (mostly dictionaries) for a large number of languages are the\nASJP database\n,\nPanLex\n, and\nBabelNet\n.\nFinally, while CLWEs underperform deep multilingual models on more challenging benchmarks (\nArtetxe et al., 2019\n), they are still preferred for easier tasks such as document classification due to their efficiency and ease of use. In addition, they serve as a useful initialisation to kick-start the training of deep models (\nMulcaire et al., 2019\n;\nAnonymous et al., 2019\n) and of unsupervised NMT models (\nArtetxe et al., 2019\n). Consequently, they may be a useful foundation for the development of more resource and parameter-efficient deep multilingual models.\nCitation\nIf you found this post helpful, consider citing\nthe tutorial\nas:\n@inproceedings{ruder2019unsupervised,\n title={Unsupervised Cross-Lingual Representation Learning},\n author={Ruder, Sebastian and S{\\o}gaard, Anders and Vuli{\\'c}, Ivan}, \n booktitle={Proceedings of ACL 2019, Tutorial Abstracts},\n pages={31--38},\n year={2019}\n}", "metadata": { "title": "Unsupervised Cross-lingual Representation Learning", "description": "This post expands on the ACL 2019 tutorial on Unsupervised Cross-lingual Representation Learning. It highlights key insights and takeaways and provides updates based on recent work, particularly unsupervised deep multilingual models.", "keywords": null, "author": null, "og:site_name": "ruder.io", "og:type": "article", "og:title": "Unsupervised Cross-lingual Representation Learning", "og:description": "This post expands on the ACL 2019 tutorial on Unsupervised Cross-lingual Representation Learning. It highlights key insights and takeaways and provides updates based on recent work, particularly unsupervised deep multilingual models.", "og:url": "https://www.ruder.io/unsupervised-cross-lingual-learning/", "og:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2019/11/unsupervised_multilingual_overview.png", "og:image:width": "1200", "og:image:height": "732", "twitter:card": "summary_large_image", "twitter:title": "Unsupervised Cross-lingual Representation Learning", "twitter:description": "This post expands on the ACL 2019 tutorial on Unsupervised Cross-lingual Representation Learning. It highlights key insights and takeaways and provides updates based on recent work, particularly unsupervised deep multilingual models.", "twitter:url": "https://www.ruder.io/unsupervised-cross-lingual-learning/", "twitter:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2019/11/unsupervised_multilingual_overview.png", "twitter:label1": "Written by", "twitter:data1": "Sebastian Ruder", "twitter:label2": "Filed under", "twitter:data2": "cross-lingual, transfer learning, natural language processing", "twitter:site": "@seb_ruder", "article:published_time": "2019-10-26T11:35:18.000Z", "article:modified_time": "2019-11-09T08:57:07.000Z", "article:tag": "natural language processing" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/fa35ac2d4b0c/code/body-000.txt", "chars": 265, "language": "text" } ], "images": [ { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2019/11/unsupervised_multilingual_overview.png", "bytes": 415458, "reused": true, "local_path": "attachments/fa35ac2d4b0c/images/001.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2019/11/unsupervised_multilingual_overview.png", "alt": "Unsupervised Cross-lingual Representation Learning", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/unsupervised_multilingual_overview.png", "bytes": 436044, "reused": true, "local_path": "attachments/fa35ac2d4b0c/images/005.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/unsupervised_multilingual_overview.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/cross_lingual_learning_taxonomy.png", "bytes": 147890, "reused": true, "local_path": "attachments/fa35ac2d4b0c/images/006.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/cross_lingual_learning_taxonomy.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/nlp_resource_hierarchy.png", "bytes": 324823, "reused": true, "local_path": "attachments/fa35ac2d4b0c/images/007.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/nlp_resource_hierarchy.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/transfer_learning_formula.png", "bytes": 154022, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-f56e7a1b5ead5d47dae4.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/transfer_learning_formula.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/projection_framework.png", "bytes": 206312, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-6ba32c1725abdfbc8828.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/projection_framework.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/mapping_visualisation.png", "bytes": 246443, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-ef293119c181c11e3afa.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/mapping_visualisation.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/self-learning_loop_wide.png", "bytes": 170229, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-356e99fb0f9a9e80c07f.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/self-learning_loop_wide.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/cross-lingual_framework_landscape.png", "bytes": 222343, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-b817ac0fa74d44686a52.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/cross-lingual_framework_landscape.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/adversarial_mapping.png", "bytes": 86969, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-b26c517a1aa42decec80.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/adversarial_mapping.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/intralingual_similarity_distributions.png", "bytes": 133339, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-a08fb13f3c6e626f446f.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/10/intralingual_similarity_distributions.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/bli_performance_vs_seed_dict_size_landscape.png", "bytes": 132427, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-a563781d12c549d738bb.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/bli_performance_vs_seed_dict_size_landscape.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/translation_language_modelling.png", "bytes": 192983, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-f0245f5b65731ef76a20.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/translation_language_modelling.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/contextual_aligned_landscape.png", "bytes": 524283, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-54e8a505d2f1e955ecfe.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/contextual_aligned_landscape.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/monolingual_transfer.png", "bytes": 410433, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-34e662def04f76c731b0.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/monolingual_transfer.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/dictionary_frequency_ranks_landscape.png", "bytes": 159349, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-73b06fed2016c905d8f2.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/dictionary_frequency_ranks_landscape.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/nn_graphs_freq_words_landscape.png", "bytes": 51485, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-e3108e5db081c6f04983.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/nn_graphs_freq_words_landscape.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/multifit_bootstrapping_landscape.png", "bytes": 219202, "reused": false, "local_path": "attachments/fa35ac2d4b0c/images/body-ff40ffba3c081258f464.png", "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2019/11/multifit_bootstrapping_landscape.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://www.ruder.io/tag/cross-lingual/", "label": "cross-lingual", "type": "a" }, { "href": "https://www.ruder.io/author/sebastian/", "label": "", "type": "a" }, { "href": "https://tinyurl.com/xlingual?ref=ruder.io", "label": "ACL 2019 tutorial on Unsupervised Cross-lingual Representation Learning", "type": "a" }, { "href": "https://sites.google.com/site/ivanvulic/?ref=ruder.io", "label": "Ivan Vulić", "type": "a" }, { "href": "https://anderssoegaard.github.io/?ref=ruder.io", "label": "Anders Søgaard", "type": "a" }, { "href": "https://docs.google.com/presentation/d/1miPu0hkBFSRTAxP-Psg0xXOvAbVSJPhQXehMQdUI88I/edit?usp=sharing&ref=ruder.io", "label": "available online", "type": "a" }, { "href": "https://ruder.io/thesis/neural_transfer_learning_for_nlp.pdf?ref=ruder.io", "label": "Ruder, 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1008.0716?ref=ruder.io", "label": "Prettenhofer & Stein, 2011", "type": "a" }, { "href": "https://pdfs.semanticscholar.org/f08f/269b715c31bfa02dc4df31f5561fd74222f0.pdf?ref=ruder.io", "label": "Wan et al., 2011", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Loanword?ref=ruder.io", "label": "loanwords", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Cognate?ref=ruder.io", "label": "cognates", "type": "a" }, { "href": "https://universaldependencies.org/?ref=ruder.io", "label": "structure of sentences", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Semantic_primes?ref=ruder.io", "label": "meaning of words", "type": "a" }, { "href": "https://wals.info/languoid?ref=ruder.io", "label": "World Atlas of Language Structures", "type": "a" }, { "href": "https://docs.google.com/presentation/d/1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc/edit?ref=ruder.io", "label": "broader transfer learning landscape", "type": "a" }, { "href": "https://www.jair.org/index.php/jair/article/view/11640/26511?ref=ruder.io", "label": "this survey", "type": "a" }, { "href": "https://www.ruder.io/state-of-transfer-learning-in-nlp/", "label": "this recent post", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Linguistic_relativity?ref=ruder.io", "label": "Sapir-Whorf hypothesis", "type": "a" }, { "href": "http://labs.theguardian.com/digital-language-divide/?ref=ruder.io", "label": "digital language divide", "type": "a" }, { "href": "https://www.washingtonpost.com/graphics/2018/business/alexa-does-not-understand-your-accent/?ref=ruder.io", "label": "occurrences", "type": "a" }, { "href": "https://www.wired.com/2017/03/voice-is-the-next-big-platform-unless-you-have-an-accent/?ref=ruder.io", "label": "of this", "type": "a" }, { "href": "https://qz.com/1141122/google-translates-gender-bias-pairs-he-with-hardworking-and-she-with-lazy-and-other-examples/?ref=ruder.io", "label": "already", "type": "a" }, { "href": "https://journals.sagepub.com/doi/10.1068/a44674?ref=ruder.io", "label": "today", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Aragonese_language?ref=ruder.io", "label": "Aragonese", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Waray_language?ref=ruder.io", "label": "Waray-Waray", "type": "a" }, { "href": "https://meta.wikimedia.org/wiki/List_of_Wikipedias_by_speakers_per_article?ref=ruder.io", "label": "disproportionally", "type": "a" }, { "href": "https://meta.wikimedia.org/wiki/List_of_Wikipedias?ref=ruder.io", "label": "large Wikipedia", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Latvian_language?ref=ruder.io", "label": "Latvian", "type": "a" }, { "href": "https://www.statmt.org/europarl/?ref=ruder.io", "label": "parallel data", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Xhosa_language?ref=ruder.io", "label": "Xhosa", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Zulu_language?ref=ruder.io", "label": "Zulu", "type": "a" }, { "href": "https://arxiv.org/pdf/1401.5694.pdf?ref=ruder.io", "label": "Padó & Lapata, 2009", "type": "a" }, { "href": "https://arxiv.org/abs/1707.02483?ref=ruder.io", "label": "Ni et al., 2017", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P19-1172/?ref=ruder.io", "label": "Nicolai & Yarowsky, 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D18-1061/?ref=ruder.io", "label": "Plank & Agić, 2018", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P16-1133/?ref=ruder.io", "label": "Zhou et al., 2016", "type": "a" }, { "href": "https://arxiv.org/abs/1807.08998?ref=ruder.io", "label": "Eger et al., 2018", "type": "a" }, { "href": "https://ai.googleblog.com/2019/10/exploring-massively-multilingual.html?ref=ruder.io", "label": "massively multilingual", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D18-1549/?ref=ruder.io", "label": "Lample et al., 2018", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P19-1019/?ref=ruder.io", "label": "Artetxe et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D18-1269/?ref=ruder.io", "label": "Conneau et al., 2018", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D19-1632/?ref=ruder.io", "label": "Guzmán et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P19-1227/?ref=ruder.io", "label": "Liu et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/L18-1344/?ref=ruder.io", "label": "Akbik & Vollgraf, 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1902.01313?ref=ruder.io", "label": "Artetxe et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1901.07291?ref=ruder.io", "label": "Lample & Conneau, 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1911.02116?ref=ruder.io", "label": "Conneau et al., 2019", "type": "a" }, { "href": "https://www.morganclaypool.com/doi/abs/10.2200/S00920ED2V01Y201904HLT042?ref=ruder.io", "label": "book", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P19-1070/?ref=ruder.io", "label": "Glavaš et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1309.4168?ref=ruder.io", "label": "Mikolov et al., 2013", "type": "a" }, { "href": "https://www.aclweb.org/anthology/N15-1104/?ref=ruder.io", "label": "Xing et al., 2015", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D16-1250/?ref=ruder.io", "label": "Artetxe et al., 2016", "type": "a" }, { "href": "https://link.springer.com/article/10.1007/BF02289451?ref=ruder.io", "label": "Schönemann, 1966", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P17-1042/?ref=ruder.io", "label": "Artetxe et al., 2017", "type": "a" }, { "href": "https://docs.google.com/presentation/d/1miPu0hkBFSRTAxP-Psg0xXOvAbVSJPhQXehMQdUI88I/edit?ref=ruder.io", "label": "self-learning procedure", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P16-1024/?ref=ruder.io", "label": "Vulić et al., 2016", "type": "a" }, { "href": "https://openreview.net/forum?id=H196sainb&ref=ruder.io", "label": "Lample et al., 2018", "type": "a" }, { "href": "https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/download/16935/16781?ref=ruder.io", "label": "Artetxe et al., 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1909.01638?ref=ruder.io", "label": "Vulić et al., 2019", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Generative_adversarial_network?ref=ruder.io", "label": "generative adversarial networks (GANs)", "type": "a" }, { "href": "https://arxiv.org/abs/1710.04087?ref=ruder.io", "label": "Conneau et al. (2018)", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P18-1072/?ref=ruder.io", "label": "Søgaard et al., 2018", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P18-1073/?ref=ruder.io", "label": "Artetxe et al. 2018", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D19-1449/?ref=ruder.io", "label": "Vulić et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D18-1056/?ref=ruder.io", "label": "Hartmann et al., 2018", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D18-1047?ref=ruder.io", "label": "Nakashole, 2018)", "type": "a" }, { "href": "https://arxiv.org/pdf/1906.01622.pdf?ref=ruder.io", "label": "Zhang et al., 2019)", "type": "a" }, { "href": "https://www.aclweb.org/anthology/N19-1188/?ref=ruder.io", "label": "Heymann et al., 2019", "type": "a" }, { "href": "https://github.com/coastalcph/stepbystep/blob/master/poster.pdf?ref=ruder.io", "label": "Hartmann et al., 2019", "type": "a" }, { "href": "https://github.com/google-research/bert/blob/master/multilingual.md?ref=ruder.io", "label": "multilingual BERT", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P19-1493/?ref=ruder.io", "label": "Pires et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D19-1077/?ref=ruder.io", "label": "Wu & Dredze, 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1910.11856?ref=ruder.io", "label": "Artetxe et al., 2019", "type": "a" }, { "href": "https://openreview.net/forum?id=HJeT3yrtDr&ref=ruder.io", "label": "Anonymous et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1911.01464?ref=ruder.io", "label": "Wu et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1909.00964?ref=ruder.io", "label": "Huang et al., 2019", "type": "a" }, { "href": "https://openreview.net/forum?id=r1xCMyBtPS&ref=ruder.io", "label": "Anonymous, 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/N19-1392/?ref=ruder.io", "label": "Mulcaire et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/N19-1162/?ref=ruder.io", "label": "Schuster et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/N19-1391/?ref=ruder.io", "label": "Aldarmaki & Diab, 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/K19-1004/?ref=ruder.io", "label": "Liu et al. (2019)", "type": "a" }, { "href": "https://openreview.net/forum?id=S1l-C0NtwS&ref=ruder.io", "label": "Anonymous et al., 2019", "type": "a" }, { "href": "https://openreview.net/forum?id=Bkle6T4YvB&ref=ruder.io", "label": "Anonymous et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D19-1090/?ref=ruder.io", "label": "Czarnowska et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D19-1328/?ref=ruder.io", "label": "Kementchedjhieva et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1909.02855?ref=ruder.io", "label": "Czarnowska et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/C12-1089/?ref=ruder.io", "label": "Klementiev et al., 2012", "type": "a" }, { "href": "https://www.aclweb.org/anthology/N18-2017/?ref=ruder.io", "label": "Gururangan et al., 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1910.07475?ref=ruder.io", "label": "Lewis et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P18-1072?ref=ruder.io", "label": "Søgaard et al., 2018", "type": "a" }, { "href": "http://nlp.fast.ai/classification/2019/09/10/multifit.html?ref=ruder.io", "label": "zero-shot transfer is not free", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D19-1572/?ref=ruder.io", "label": "Eisenschlos et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/D19-1374/?ref=ruder.io", "label": "Tsai et al., 2019", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P19-1310/?ref=ruder.io", "label": "Agić & Vulić, 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1907.05791?ref=ruder.io", "label": "Schwenk et al., 2019", "type": "a" }, { "href": "https://asjp.clld.org/?ref=ruder.io", "label": "ASJP database", "type": "a" }, { "href": "https://panlex.org/?ref=ruder.io", "label": "PanLex", "type": "a" }, { "href": "https://babelnet.org/?ref=ruder.io", "label": "BabelNet", "type": "a" }, { "href": "https://www.aclweb.org/anthology/P19-4007?ref=ruder.io", "label": "the tutorial", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:52:00Z", "source_url": "https://www.ruder.io/unsupervised-cross-lingual-learning/", "papers": [ { "status": "http_404", "url": "https://ruder.io/thesis/neural_transfer_learning_for_nlp.pdf?ref=ruder.io", "retried_at": "2026-10-20T17:40:56Z" }, { "status": "ok", "url": "https://arxiv.org/pdf/1008.0716", "bytes": 708733, "reused": false, "local_path": "papers/repair/8f6ba7e96ada6ed2f5b5.pdf" }, { "status": "http_202", "url": "https://pdfs.semanticscholar.org/f08f/269b715c31bfa02dc4df31f5561fd74222f0.pdf?ref=ruder.io", "retried_at": "2026-10-20T17:40:57Z" }, { "status": "ok", "url": "https://arxiv.org/pdf/1401.5694.pdf?ref=ruder.io", "bytes": 334387, "reused": false, "local_path": "papers/repair/5afd9033199117ccc193.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1707.02483", "bytes": 497942, "reused": false, "local_path": "papers/repair/fa18951d7c7a63870dca.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1807.08998", "bytes": 300741, "reused": false, "local_path": "papers/repair/09c6c08e47286ad8fec8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1902.01313", "bytes": 282862, "reused": false, "local_path": "papers/repair/aa8350b33634f842c779.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1901.07291", "bytes": 342326, "reused": false, "local_path": "papers/repair/5f13ba6b638f9c631c3d.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1911.02116", "bytes": 398981, "reused": true, "local_path": "papers/repair/069e86d8c4407636cd2f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1309.4168", "bytes": 287225, "reused": false, "local_path": "papers/repair/ab2b3d11a7746fa2d132.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1909.01638", "bytes": 1014978, "reused": false, "local_path": "papers/repair/96d9ee5dae4d89d76a8e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1710.04087", "bytes": 634800, "reused": false, "local_path": "papers/repair/521656494999c046b4b9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1906.01622.pdf?ref=ruder.io", "bytes": 1197995, "reused": false, "local_path": "papers/repair/d8ab5aa47a9b306f0023.pdf" }, { "status": "ok", "url": "https://raw.githubusercontent.com/coastalcph/stepbystep/master/poster.pdf", "bytes": 310791, "reused": false, "local_path": "papers/repair/49499510964094c49b24.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1910.11856", "bytes": 614586, "reused": false, "local_path": "papers/repair/b6528b5b8ccdd3ecb04b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1911.01464", "bytes": 712704, "reused": false, "local_path": "papers/repair/018f96bf306b0f3f658f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1909.00964", "bytes": 3555680, "reused": false, "local_path": "papers/repair/4721101e0d653269b646.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1909.02855", "bytes": 487835, "reused": false, "local_path": "papers/repair/db27df3217c39af19bdb.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1910.07475", "bytes": 1283754, "reused": false, "local_path": "papers/repair/e62a3c10f65f8d251961.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1907.05791", "bytes": 412170, "reused": true, "local_path": "papers/repair/6d9d1084cc9061d161d7.pdf" } ], "actual_video_files": [], "body_chars": 32048, "expected_body_images": 18, "discovery_status": "ok", "delivery_status": "partial", "missing_reason": "http_202；http_404", "raw_html": "attachments/fa35ac2d4b0c/pages/source.html", "local_html": "attachments/fa35ac2d4b0c/pages/article.html", "body_base_path": "attachments/fa35ac2d4b0c/pages/", "body_markdown_path": "attachments/fa35ac2d4b0c/pages/article.md" }