{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Sebastian Ruder",
  "title_listed": "True Zero-shot MT",
  "keywords": "零样本,机器翻译,长上下文",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.ruder.io/true-zero-shot-mt/",
  "publish_date": "2024-02",
  "corpus_stage": "推理部署",
  "notes": "真正的零样本机器翻译探讨",
  "id": "375117a4d75e",
  "collected_at": "2026-10-02T14:15:51",
  "title": "True Zero-shot MT",
  "body_markdown": "multilingual\nTrue Zero-shot MT\nThis post discusses recent results on extremely long-context benchmarks, explores true zero-shot machine translation (MT), and considers how to teach LLMs a new language like humans.\nSebastian Ruder\n27 Feb 2024\n•\n8 min read\nShare\nPhD thesis\nof Eline Visser (co-author of\nMTOB\n)\nLittle over a week ago,\nGemini 1.5 reported\nclose to human-level performance on\nMTOB\n, a recent challenging translation dataset. In this post, we’ll dig into this result, explore true zero-shot machine translation (MT), and consider how to teach LLMs a new language like humans.\nThis post was first published in\nNLP News\n.\nLow-resource MT\nTo set the scene, let’s first consider what it means for a language to be considered “low-resource”. As with LLMs, the performance of MT models depends on the amount of training data—both parallel and monolingual—available in a given language. As a result, there is a\ngulf between languages with lots of data and languages with little data\nin common pre-training corpora. The latter are typically referred to as “low-resource”.\nTo bridge the gap between resource-rich and resource-poor languages and make machine translation more accessible, new translation benchmarks have been created that cater specifically to low-resource languages. The\nConference on Machine Translation (WMT)\nnow regularly hosts shared tasks on low-resource MT such as for\nIndic\nand\nAfrican languages\n; workshops such as\nAmericasNLP\nsupport\nindigenous languages\n; and large-scale decentralized collaborations such as\nMasakhane\n,\nSEACrowd\nand\nAya\ncreated MT datasets for\nAfrican languages\n,\nIndonesian languages\n, and\n100+ languages\nrespectively. Recently,\nFLORES-200\nexpands translation data coverage to 200 languages. Beyond theses efforts, through extensive work on data cleaning, filtering, and language identification, researchers have been able to obtain data and train MT models for 1000+ languages (\nBapna et al., 2022\n;\nNLLB Team, 2022\n).\nLLMs are typically trained on parallel data (\nKale et al., 2021\n) and are increasingly used for translation (\nVilar et al., 2023\n). However, in light of ever-larger pre-training datasets, the opacity of pre-training data, and challenges of language identification on the web (\nCaswell et al., 2020\n), it is unclear how much data LLMs have seen in a low-resource language during pre-training. Chances are that most LLMs have seen\nsome\ndata in most languages that are available on the web.\nFrom an experimental perspective, it is thus not straightforward to assess how much data LLMs\nactually\nneed to learn to translate in a new language. While we could restrict the languages a model is trained on and examine its performance on a held-out language with increasing amounts of training data, we know that most LLM capabilities only emerge at scale. So how can we study the translation abilities of fully pre-trained LLMs in a controlled setting?\nTrue Zero-shot MT\nTo study this setting, we have to look at translating a language that was\ntruly unseen\nduring pre-training. For simplicity, I use the term\ntrue zero-shot MT\nto designate the setting of translating to a language with no pre-training data using only in-context learning data.\nWhile we can find data for 1500+ languages on the web (\nBapna et al., 2022\n), there are around 7000+ languages spoken around the world. So the remaining 5500 languages with little or no presence online are potential candidates for our target language.\nAs expected, for these languages there is very little data available that has been used in standard LLM pipelines. Interestingly, the resources that\nare\navailable for these languages are similar to those humans might use to learn a\nsecond language (L2)\nincluding:\na list of words and their translations to learn a language’s vocabulary;\npaired sentences to learn about word usage, word order, and some morphology;\na grammar book to study the structure of the language.\nLet’s take a look at how these resources can be used by LLMs:\n1. A Bilingual Word List\nDatabase coverage of bilingual word lists in\nPanLex\n.\nWhile lists of words and their translation only teach us a limited amount about a new language, they are available for a huge number of languages thanks to projects such as\nPanLex\n. However, for many languages, they only contain translation pairs for a small number of terms such as the numbers from 1–10.\nBilingual lexicons have been a core resource for aligning word embeddings across languages (\nRuder et al., 2019\n). They also have been used to bootstrap unsupervised MT systems (\nArtetxe et al., 2018\n), though, translating text word-by-word does not get you far, even if you account for word order differences (\nLample et al., 2018\n).\nOn the other hand, even in the best LLMs, cross-lingual alignment of words in the vocabulary is still not perfect. Bilingual lexicons are thus a potential viable—and large-coverage—form of supervision. There are multiple ways in which they can be used with LLMs:\ndata augmentation\n: “noisy” code-mixed samples are created by replacing words in source language sentences with their target language translations (\nWang et al., 2022\n,\nReid & Artetxe, 2022\n);\nlexical prompting\n: prepending\n<source word, translation>\npairs to the prompt for source words that occur in the input (\nGhazvininejad et al., 2023\n);\nparallel data\n:\n<source word, translation>\npairs are treated as “sentence” pairs and added to the existing parallel data.\nJones et al. (2023)\ncompare several augmentations. They find that they provide similar gains for unsupervised machine translation and that high-quality bilingual lexicons such as\nGATITOS\nare crucial. Recently,\nKoto et al. (2024)\nuse PanLex to extend sentiment lexicons to more languages while\nYong et al. (2024)\ngenerate data that maximizes lexical overlap with downstream tasks.\n2. Few Parallel Sentences\nParallel data is the bread and butter of MT research. MT models and LLMs are trained on millions of parallel sentences. Prior work on low-resource MT such as for Nepalese–English used 100,000s of parallel sentences (\nGuzmán et al., 2019\n).\nMore recent work studied the impact of around 1–6k professionally translated sentences across different low-resource languages (\nMaillard et al., 2023\n). They find that already 1,000s of high-quality parallel sentences are helpful while multilingual training and back-translation are crucial to achieve good performance.\nBut what can you do if you only have very few parallel sentences for a language? Such scenarios are common in the form of\nRosetta Stone\npuzzles in Linguistic Olympiads.\nAn example\nRosetta Stone\npuzzle\n(Sahin et al., 2020)\n.\nThese puzzles are designed so that only a minimum number of parallel sentences are required to deduce the relevant translations to and from the target language (Chickasaw above).\nSahin et al. (2020)\ncollected 100 such puzzles. LLMs such as GPT-3.5 Turbo still fall short of solving these puzzles with an Exact Match score of around 20% and even advanced prompting methods are not helpful (\nLin et al., 2023\n).\nBean et al. (2024)\nfurther collected 1,133 problems into a linguistic reasoning benchmark. They observe that models perform poorly on the higher difficulty problems, with the best model only achieving 38.7% accuracy.\n3. A Grammar Book\nWhile both bilingual lexicons and parallel sentences are well-known data sources for training MT models, grammar books are much less common. A reference grammar is a key product of linguistic fieldwork, which can also result in other artifacts such as language learning materials including alphabet books, learners’ guides, etc.\nMany reference grammars are available as books in pdf format. As part of sourcing open-source OCR data from Google Books for\nXTREME-UP\n(\nRuder et al., 2023\n), we came across many public-domain books in under-represented languages that described the grammar or vocabulary of the language and were created as part of missionary efforts and linguistic fieldwork.\nCompared to these,\nA grammar of Kamalang\n(\nVisser, 2022\n) used in MTOB is a more recent reference grammar. Reference grammars are\npublicly available\nin\n100s of languages\nand present a promising and—with exception of MTOB—untapped resource for studying the language acquisition of LLMs.\nAn excerpt from\nA grammar of Kamalang\n.\n4. Putting Everything Together: MTOB\nMTOB (Machine Translation from One Book;\nTanzer et al., 2023\n) is a recent dataset that provides the three above resources for\nKalamang\n, an endangered language spoken by less than 200 people, which is essentially absent from pre-training corpora.\nIn addition, Kalamang is not closely related to other languages with many speakers (which is important to measure true zero-shot performance) and uses the Latin script, which makes it easy to process with LLMs. The authors obtained the permission of the Kalamang-speaking community for using their data, which is crucial for this type of work.\nCompared to standard MT evaluation where native speaker translations are the benchmark to beat, producing native-level translations may not be feasible using only the provided resources. The authors thus provide another human baseline: The first author Garrett Tanzer learned how to translate Kalamang from scratch by reading the grammar book for 10+ hours and then used the parallel data and the Internet as reference when performing the translation task over the course of several weeks.\nThat\nis dedication to human evaluation!\nIn their evaluation, all LLMs underperform the human baseline, with Claude 2 performing best in comparison. Recently,\nGemini 1.5 Pro\nusing its extremely long context window is able to use in-context learning on the Kalamang resources to improve substantially on the English->Kalamang translation task. Let’s look at what these results mean for the field overall and for under-represented languages in particular.\nFuture Impact\nLong Context Modeling\nIt’s quite remarkable how long-context models have progressed over the last years. Being able to translate into a new language using only in-context learning based on existing linguistic resources is an impressive feat. Nevertheless, parameter-efficient fine-tuning and retrieval-augmented generation (RAG) are important baselines for this setting that can put the in-context learning performance in perspective.\nLong Context Datasets\nWhile many challenging long-context benchmarks are synthetic in nature, including the most challenging tasks in Long Range Arena (\nTay et al., 2021\n) and the needle-in-the-haystack tasks in the Gemini 1.5 evaluation, datasets like MTOB provide a more realistic setting using standard linguistic resources. In addition, they enable comparison to a human baseline and to a (potentially unattainable) human native speaker. Given the increasing popularity of long-context models, I hope we see more long-context datasets grounded in realistic data and human-level comparisons.\nUnder-represented Languages\nWhile these results highlight the potential of LLMs to learn to translate with very little data, many under-represented languages are primarily spoken; they do not have a written tradition or a standardized orthography. Text-based NLP technology is thus of limited use to them. For these languages, multi-modal LLMs will be an important foundation. Given the powerful capabilities of LLMs, we should design evaluations with the needs of the language communities in mind. A promising area is to support ‘contact languages’ including creoles and regional varieties of standardized languages (\nBird, 2022\n).\nNLP and Cognitive Science\nIn order to study language acquisition in language models, benchmarks such as the\nBabyLM Challenge\nemploy a developmentally plausible corpus including mostly transcribed speech and a limited number of tokens. However, the embodied, interactive, and multi-modal nature of\nfirst language (L1)\nacquisition is challenging to replicate with current models. L2 acquisition in the form of true zero-shot MT may be a more accessible testbed to study how a model learns a new language based on limited linguistic resources.\nInterpretability and Model Understanding\nGrammar books and language learning resources similarly provide a means to analyze how LLMs acquire new information and how they use their existing knowledge to reason over new inputs. They can be used to better understand models’ inner workings via tasks such as grammar induction (\nKim et al., 2020\n). For instance, an interesting question is whether the sentences a model retrieves via RAG are similar to those consulted by a human when translating into a new language.\nNLP and Linguistics\nTo make most use of such resources, both in obtaining and understanding the data as well as interpreting model results requires collaborating with linguists. MTOB provides an example of how such a collaboration can look like in practice, with the linguist actively participating in the research and co-authoring the paper. Such inter-disciplinary collaborations, while challenging and complex, are often a breath of fresh air—so I hope to see more of them in the future.",
  "metadata": {
    "title": "True Zero-shot MT",
    "description": "This post discusses recent results on extremely long-context benchmarks, explores true zero-shot machine translation (MT), and considers how to teach LLMs a new language like humans.",
    "keywords": null,
    "author": null,
    "og:site_name": "ruder.io",
    "og:type": "article",
    "og:title": "True Zero-shot MT",
    "og:description": "This post discusses recent results on extremely long-context benchmarks, explores true zero-shot machine translation (MT), and considers how to teach LLMs a new language like humans.",
    "og:url": "https://www.ruder.io/true-zero-shot-mt/",
    "og:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/grammar_kalamang.png",
    "og:image:width": "1169",
    "og:image:height": "703",
    "twitter:card": "summary_large_image",
    "twitter:title": "True Zero-shot MT",
    "twitter:description": "This post discusses recent results on extremely long-context benchmarks, explores true zero-shot machine translation (MT), and considers how to teach LLMs a new language like humans.",
    "twitter:url": "https://www.ruder.io/true-zero-shot-mt/",
    "twitter:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2024/10/grammar_kalamang.png",
    "twitter:label1": "Written by",
    "twitter:data1": "Sebastian Ruder",
    "twitter:label2": "Filed under",
    "twitter:data2": "multilingual, translation",
    "twitter:site": "@seb_ruder",
    "article:published_time": "2024-02-27T15:58:00.000Z",
    "article:modified_time": "2024-10-24T09:54:40.000Z",
    "article:tag": "translation"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2024/10/grammar_kalamang.png",
      "bytes": 1051326,
      "reused": true,
      "local_path": "attachments/375117a4d75e/images/001.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2024/10/grammar_kalamang.png",
      "alt": "True Zero-shot MT",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd92e197c-ae1e-4709-b629-1117b2448496_2204x378.png",
      "bytes": 24762,
      "reused": true,
      "local_path": "attachments/375117a4d75e/images/005.png",
      "src": "https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd92e197c-ae1e-4709-b629-1117b2448496_2204x378.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf229425-88b5-4f0b-8755-2754f1b71671_2558x1128.png",
      "bytes": 68345,
      "reused": true,
      "local_path": "attachments/375117a4d75e/images/006.png",
      "src": "https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf229425-88b5-4f0b-8755-2754f1b71671_2558x1128.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1ffbde3-d3fc-49b9-a596-ba4ca2c0ae4c_1958x914.png",
      "bytes": 223048,
      "reused": true,
      "local_path": "attachments/375117a4d75e/images/007.png",
      "src": "https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1ffbde3-d3fc-49b9-a596-ba4ca2c0ae4c_1958x914.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.ruder.io/tag/multilingual/",
      "label": "multilingual",
      "type": "a"
    },
    {
      "href": "https://www.ruder.io/author/sebastian/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://lucris.lub.lu.se/ws/portalfiles/portal/87549524/Visser_Eline_2020_A_grammar_of_Kalamang_the_Papuan_language_of_the_Karas_Islands.pdf",
      "label": "PhD thesis",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2309.16575",
      "label": "MTOB",
      "type": "a"
    },
    {
      "href": "https://x.com/JeffDean/status/1758149033473020081?s=20&ref=ruder.io",
      "label": "Gemini 1.5 reported",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2309.16575?ref=ruder.io",
      "label": "MTOB",
      "type": "a"
    },
    {
      "href": "https://newsletter.ruder.io/p/true-zero-shot-mt?ref=ruder.io",
      "label": "NLP News",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.acl-main.560/?ref=ruder.io",
      "label": "gulf between languages with lots of data and languages with little data",
      "type": "a"
    },
    {
      "href": "https://www2.statmt.org/wmt23/?ref=ruder.io",
      "label": "Conference on Machine Translation (WMT)",
      "type": "a"
    },
    {
      "href": "https://www2.statmt.org/wmt23/indic-mt-task.html?ref=ruder.io",
      "label": "Indic",
      "type": "a"
    },
    {
      "href": "https://www2.statmt.org/wmt23/african-mt-task.html?ref=ruder.io",
      "label": "African languages",
      "type": "a"
    },
    {
      "href": "https://turing.iimas.unam.mx/americasnlp/2023_st.html?ref=ruder.io",
      "label": "AmericasNLP",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2023.americasnlp-1.23/?ref=ruder.io",
      "label": "indigenous languages",
      "type": "a"
    },
    {
      "href": "https://www.masakhane.io/?ref=ruder.io",
      "label": "Masakhane",
      "type": "a"
    },
    {
      "href": "https://github.com/SEACrowd/seacrowd-datahub?ref=ruder.io",
      "label": "SEACrowd",
      "type": "a"
    },
    {
      "href": "https://cohere.com/research/aya?ref=ruder.io",
      "label": "Aya",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.findings-emnlp.195/?ref=ruder.io",
      "label": "African languages",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2023.eacl-main.57/?ref=ruder.io",
      "label": "Indonesian languages",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2402.06619?ref=ruder.io",
      "label": "100+ languages",
      "type": "a"
    },
    {
      "href": "https://github.com/facebookresearch/flores/blob/main/flores200/README.md?ref=ruder.io",
      "label": "FLORES-200",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2205.03983?ref=ruder.io",
      "label": "Bapna et al., 2022",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2207.04672?ref=ruder.io",
      "label": "NLLB Team, 2022",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.acl-short.87/?ref=ruder.io",
      "label": "Kale et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2023.acl-long.859/?ref=ruder.io",
      "label": "Vilar et al., 2023",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.coling-main.579/?ref=ruder.io",
      "label": "Caswell et al., 2020",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Second_language?ref=ruder.io",
      "label": "second language (L2)",
      "type": "a"
    },
    {
      "href": "https://panlex.org/?ref=ruder.io",
      "label": "PanLex",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.04902?ref=ruder.io",
      "label": "Ruder et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1710.11041?ref=ruder.io",
      "label": "Artetxe et al., 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1711.00043?ref=ruder.io",
      "label": "Lample et al., 2018",
      "type": "a"
    },
    {
      "href": "http://expanding%20pretrained%20models%20to%20thousands%20more%20languages%20via%20lexicon-based%20adaptation/",
      "label": "Wang et al., 2022",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2022.naacl-main.58/?ref=ruder.io",
      "label": "Reid & Artetxe, 2022",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2302.07856?ref=ruder.io",
      "label": "Ghazvininejad et al., 2023",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2023.emnlp-main.26/?ref=ruder.io",
      "label": "Jones et al. (2023)",
      "type": "a"
    },
    {
      "href": "https://github.com/google-research/url-nlp/tree/main/gatitos?ref=ruder.io",
      "label": "GATITOS",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2402.02113?ref=ruder.io",
      "label": "Koto et al. (2024)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2402.14086?ref=ruder.io",
      "label": "Yong et al. (2024)",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/D19-1632/?ref=ruder.io",
      "label": "Guzmán et al., 2019",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.acl-main.115/?ref=ruder.io",
      "label": "(Sahin et al., 2020)",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2023.rocling-1.33/?ref=ruder.io",
      "label": "Lin et al., 2023",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.06196?ref=ruder.io",
      "label": "Bean et al. (2024)",
      "type": "a"
    },
    {
      "href": "https://github.com/google-research/xtreme-up?ref=ruder.io",
      "label": "XTREME-UP",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2023.findings-emnlp.125/?ref=ruder.io",
      "label": "Ruder et al., 2023",
      "type": "a"
    },
    {
      "href": "https://langsci-press.org/catalog/book/344?ref=ruder.io",
      "label": "Visser, 2022",
      "type": "a"
    },
    {
      "href": "https://guides.tricolib.brynmawr.edu/c.php?g=285265&p=1900305&ref=ruder.io",
      "label": "publicly available",
      "type": "a"
    },
    {
      "href": "https://linguistic-typology.org/grammarwatch/all/by-macro-area/?ref=ruder.io",
      "label": "100s of languages",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Kalamang_language?ref=ruder.io",
      "label": "Kalamang",
      "type": "a"
    },
    {
      "href": "https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf?ref=ruder.io",
      "label": "Gemini 1.5 Pro",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=qVyeW-grC2k&ref=ruder.io",
      "label": "Tay et al., 2021",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2022.acl-long.539/?ref=ruder.io",
      "label": "Bird, 2022",
      "type": "a"
    },
    {
      "href": "https://babylm.github.io/?ref=ruder.io",
      "label": "BabyLM Challenge",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/First_language?ref=ruder.io",
      "label": "first language (L1)",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=H1xPR3NtPB&ref=ruder.io",
      "label": "Kim et al., 2020",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:58Z",
  "source_url": "https://www.ruder.io/true-zero-shot-mt/",
  "papers": [
    {
      "status": "ok",
      "url": "https://lucris.lub.lu.se/ws/portalfiles/portal/87549524/Visser_Eline_2020_A_grammar_of_Kalamang_the_Papuan_language_of_the_Karas_Islands.pdf",
      "bytes": 57297533,
      "reused": false,
      "local_path": "papers/repair/0fae437e110bcc1cff3d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2309.16575",
      "bytes": 402284,
      "reused": true,
      "local_path": "papers/repair/c0d0c409fb80f4e43987.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2309.16575",
      "bytes": 402284,
      "reused": true,
      "local_path": "papers/repair/c0d0c409fb80f4e43987.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2402.06619",
      "bytes": 9546394,
      "reused": false,
      "local_path": "papers/repair/02252ac2bdd9c68bca69.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2205.03983",
      "bytes": 1380921,
      "reused": true,
      "local_path": "papers/repair/73ac0a9dea6a40108ebb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2207.04672",
      "bytes": 8094540,
      "reused": false,
      "local_path": "papers/repair/40a283dec199a57d9907.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.04902",
      "bytes": 1541564,
      "reused": false,
      "local_path": "papers/repair/c94feffbb0c004e1c452.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1710.11041",
      "bytes": 344734,
      "reused": false,
      "local_path": "papers/repair/c86d49ced0d645045c2c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1711.00043",
      "bytes": 636491,
      "reused": false,
      "local_path": "papers/repair/f718b710a4e610c6285e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2302.07856",
      "bytes": 472789,
      "reused": false,
      "local_path": "papers/repair/f8b99d3f5682c020f8e1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2402.02113",
      "bytes": 627872,
      "reused": false,
      "local_path": "papers/repair/6c87bd0f4b4df8410501.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2402.14086",
      "bytes": 914531,
      "reused": false,
      "local_path": "papers/repair/740ba6abb41daf150631.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.06196",
      "bytes": 3240501,
      "reused": false,
      "local_path": "papers/repair/a6e0b970bafe8fdc2185.pdf"
    },
    {
      "status": "ok",
      "url": "https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf?ref=ruder.io",
      "bytes": 5940002,
      "reused": false,
      "local_path": "papers/repair/80f3952e257e7199ec7e.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 13081,
  "expected_body_images": 4,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/375117a4d75e/pages/source.html",
  "local_html": "attachments/375117a4d75e/pages/article.html",
  "body_base_path": "attachments/375117a4d75e/pages/",
  "body_markdown_path": "attachments/375117a4d75e/pages/article.md"
}