{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Sebastian Ruder",
  "title_listed": "Modular Deep Learning",
  "keywords": "模块化,深度学习,适配器",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.ruder.io/modular-deep-learning/",
  "publish_date": "2023-02",
  "corpus_stage": "预训练/SFT",
  "notes": "模块化深度学习综述（adapter、LoRA等）",
  "id": "7e30313ecf1c",
  "collected_at": "2026-10-02T14:15:37",
  "title": "Modular Deep Learning",
  "body_markdown": "Modular Deep Learning\nAn overview of modular deep learning across four dimensions (computation function, routing function, aggregation function, and training setting).\nSebastian Ruder\n23 Feb 2023\n•\n11 min read\nShare\nCase studies of modular deep learning methods. From left to right:\nMAD-X\n,\nPolytropon\n,\nMixture-of-Experts Transformer\n.\nThis post gives a brief overview of modularity in deep learning. For a more in-depth review, refer to\nour survey\n. For modular fine-tuning for NLP, check out our\nEMNLP 2022 tutorial\n. For more resources, check out\nmodulardeeplearning.com\n.\nFuelled by scaling laws, state-of-the-art models in machine learning have been growing larger and larger. These models are monoliths. They are pre-trained from scratch in highly choreographed engineering endeavours. Due to their size, fine-tuning has become expensive while alternatives, such as in-context learning are often brittle in practice. At the same time, these models are still bad at many things—symbolic reasoning, temporal understanding, generating multilingual text, etc.\nModularity may help us address some of these outstanding challenges. By modularising models, we can separate fundamental knowledge and reasoning abilities about language, vision, etc from domain and task-specific capabilities. Modularity also provides a versatile way to extend models to new settings and to augment them with new abilities.\nWe give an in-depth overview of modularity in our survey on Modular Deep Learning. In the following, I will highlight some of the most important observations and takeaways related to the following concepts:\nTaxonomy\nComputation Function\nRouting Function\nAggregation Function\nTraining Setting\nPurposes of Modularity\nApplications in Transfer Learning\nFuture Directions\nConclusion\nTaxonomy\nWe categorise modular approaches based on four dimensions:\nComputation function\n: How the module is implemented.\nRouting function\n: How active modules are selected.\nAggregation function\n: How outputs of active modules are aggregated.\nTraining setting\n: How modules are trained.\nWe provide case studies of different configurations of these components below.\nCase studies of modular deep learning. Green components illustrate different\nrouting functions\n, shade-of-purple components illustrate different modular\ncomputation functions\n.\n(a)\nMAD-X\nuses adapter layers with fixed routing for zero-shot cross-lingual transfer.\n(b)\nPolytropon\nuses low-rank adapters with hard learned routing for few-shot task adaptation.\n(c)\nMoE Transformers\nuse MLPs with top-$k$ soft routing to scale to large model sizes.\nComputation Function\nWe consider a neural network $f_\\theta$ as a composition of functions $f_{\\theta_1} \\odot f_{\\theta_2} \\odot \\ldots \\odot f_{\\theta_l}$, each with their own set of parameters $\\theta_i$. A function can be a layer or a component of a layer such as a linear transformation.\nWe identify three core types of computation functions that compose a module with parameters $\\phi$ with a model’s functions:\nParameter composition.\nModules modify the model on the level of individual weights: $f_i^\\prime(\\boldsymbol{x}) = f_{\\theta_i \\oplus \\phi}(\\boldsymbol{x})$.\nInput composition.\nA function's input $\\boldsymbol{x}$ is concatenated with the module parameters: $f_i^\\prime(\\boldsymbol{x}) = f_{\\theta_i}([\\boldsymbol{x}, \\phi])$.\nFunction composition.\nThe outputs of the model's function and the module are combined: $f_i^\\prime(\\boldsymbol{x}) = f_{\\theta_i} \\odot f_{\\phi}(\\boldsymbol{x})$.\nWe provide an overview of the three comptuation functions (in addition to a hypernetwork) as part of a Transformer architecture below:\nDifferent computation functions in a Transformer architecture. Task-specific modular components are illustrated in magenta and purple.\n(a) Parameter Composition:\nA sparse sub-network in the linear layer as part of multi-head-attention.\n(b) Input Composition:\nPrefix tuning\nextends the input by prepending embeddings to key and value matrices in the Transformer layer.\n(c) Function Composition:\nTask-specific bottleneck layers that transform the hidden representations are inserted in each layer.\n(d) Hypernetwork:\nA small separate neural network generates modular parameters conditioned on metadata.\nParameter composition.\nWe identify two main ways modules are used to change the parameters of a model: a) updating a sparse subset of parameters and b) updating parameters in a low-dimensional subspace. Sparse methods are closely related to pruning and the\nlottery ticket hypothesis\n. Sparse methods can be\nstructured\nand only applied to specific parameter groups.\nInput composition.\nPrompt-based learning can be seen as finding a task-specific text prompt whose embedding $\\phi$ elicits the desired behaviour. Alternatively,\ncontinuous\nprompts can be learned directly—in the input or at each layer of a model.\nFunction composition.\nThis is the most general category. It subsumes standard multi-task learning methods, modules that adapt a pre-trained model (known as 'adapters'), and rescaling methods. In addition, parameter and input composition methods can also be expressed as function composition. For illustration, we provide examples of three function composition methods below.\nDifferent approaches of function composition.\n(a)\nSequential Bottleneck Adapter\n:\nThe first adapter architecture proposed for Transformers consisting of two bottleneck layers placed after the multi-head attention (MHA) and feed-forward (FF) layers.\n(b)\nParallel Bottleneck Adapter\n:\nBottleneck layers processed in parallel to the MHA and FF layers of the pre-trained transformer components.\n(c)\n(IA)$^3$\n:\nRescaling operations performed within the MHA and FF layers.\nModule parameter generation.\nInstead of learning module parameters directly, they can be generated using an auxiliary model (a hypernetwork) conditioned on additional information and metadata.\nWe provide a high-level overview of some of the trade-offs of the different computation functions below. Refer to the\ntutorial slides\nor the\nsurvey\nfor detailed explanations.\nParameter efficiency\nTraining efficiency\nInference efficiency\nPerformance\nCompositionality\nParameter composition\n+\n-\n++\n+\n+\nInput composition\n++\n--\n--\n-\n+\nFunction composition\n-\n+\n-\n++\n+\nRouting Function\nA routing function $r(\\cdot)$ determines which modules are active based on a given input by assigning a score $\\alpha_i$ to each module from an inventory $M$. We give an overview of the different routing methods below.\nDifferent routing methods.\n(a) Fixed Routing:\nExamples are passed to modules based on a pre-defined logic, known a priori.\n(b) Hard Learned Routing:\nLearned hard selection modules.\n(c) Soft Learned Routing:\nSoft selection and weighting of modules.\nThe routing function can be\nfixed\nand each routing decision is made based on prior knowledge about the task. Alternatively, routing can be\nlearned\n. Learned routing methods differ in whether they learn a\nhard\n, binary selection of modules or a\nsoft\nselection via a probability distribution over modules.\nFixed routing.\nFixed routing employs metadata such as the task identity to make discrete routing decisions\nbefore\ntraining. Fixed routing is used in most function composition methods such as multi-task learning and adapters. Fixed routing can select different modules for different aspects of the target setting such as task and language in NLP or robot and task in RL, which enables generalisation to unseen scenarios.\nLearned routing.\nLearned routing is typically implemented via an MLP and introduces additional challenges, including training instability, module collapse, and overfitting. Current learned routing methods are often sub-optimal as they under-utilise and under-specialise modules. However, when there is no one-to-one mapping between task and corresponding skill, they are the only option available.\nHard learned routing.\nHard learned routing models the choice of whether a module is active as a binary decision. As discrete decisions cannot be learned directly with gradient descent, methods learn hard routing via reinforcement learning, evolutionary algorithms, or stochastic re-parametrisation.\nSoft learned routing.\nSoft routing methods sidestep a discrete selection of modules by learning a weighted combination in the form of a probability distribution over available modules. A classic example is mixture of experts. As activating all modules is expensive, recent methods learn to only route the top-$k$ and even top-1 modules. Routing on the token level leads to more efficient training but limits the expressiveness of modular representations.\nLevel of routing.\nRouting can select modules globally for the entire network, make different allocations per layer, or even make hierarchical routing decisions. We illustrate different routing levels below.\nDifferent routing levels.\n(a) Layer-wise Routing:\nThe indices are chosen based on the input to the current layer.\n(b) Naive Global Routing:\nThe same indices of the modules are chosen for all the layers of the model.\n(c) Global Routing:\nThe configuration (possibly different for each layer) is chosen globally.\nAggregation Function\nThe aggregation function determines how the outputs of modules selected via routing are combined. In practice, routing and aggregation are often combined. Aggregation functions can be categorised similarly to\ncomputation functions\n; whereas computation functions compose\nshared model\nparts with\nmodule\ncomponents, aggregation functions aggregate multiple\nmodule\ncomponents on different levels:\nParameter aggregation.\nModule parameters are aggregated: $f_i^\\prime(\\boldsymbol{x}) = f_{\\boldsymbol{\\phi_i}^{1} \\oplus \\dots \\oplus \\boldsymbol{\\phi}_i^{|M|}}(\\boldsymbol{x})$.\nRepresentation aggregation.\nModular representations are aggregated: $f_i^\\prime(\\boldsymbol{x}) = f_{\\boldsymbol{\\theta}_i}(\\boldsymbol{x}) \\oplus f_{\\boldsymbol{\\phi}_i^1}(\\boldsymbol{x}) \\oplus \\dots \\oplus f_{\\boldsymbol{\\phi}_i^{|M|}}(\\boldsymbol{x})$.\nInput aggregation.\nModule parameters are concatenated at the input level: $f_i^\\prime(\\boldsymbol{x}) = f_{\\boldsymbol{\\theta_i}}([\\boldsymbol{\\phi_i^1}, \\dots, \\boldsymbol{\\phi_i^{|M|}}, \\boldsymbol{x}])$.\nFunction aggregation.\nModular functions are aggregated: $f_i^\\prime(\\boldsymbol{x}) = f_{\\boldsymbol{\\phi}_i^{1}} \\circ ... \\circ f_{\\boldsymbol{\\phi}_i^{|M|}}(\\boldsymbol{x})$.\nParameter aggregation.\nAggregating information from multiple modules by interpolating their weights is closely linked to\nlinear mode connectivity\n, which shows that under certain conditions such as the same initialisation, two networks are connected by a linear path of non-increasing error. Based on this assumption, modular edits can be performed on a model using arithmetic operations in order to remove or elicit certain information in the model.\nRepresentation aggregation.\nAlternatively, the outputs of different modules can be interpolated by aggregating the modules' hidden representations. One way to perform this aggregation is to learn a weighted sum of representations, similar to how\nrouting\nlearns a score $\\alpha_i$ per module. We can also learn a weighting that takes into account the hidden representations, such as via attention.\nInput aggregation\n.  In prompting, providing a model with multiple instructions or multiple exemplars via concatenation can be seen as a form of input aggregation. Soft prompts can be learned for different settings such as task and language or attributes and objects and aggregated via concatenation.\nFunction aggregation.\nFinally, we can aggregate modules on the function level by varying the order of computation. We can aggregate them sequentially where the output of one module is the input of the next module, etc. For more complex module configurations, we can aggregate modules hierarchically based on a tree structure.\nTraining Setting\nThe last dimension along which we can differentiate modular methods is the way they are trained. We identify three modular training strategies: 1) joint training; 2) continual learning, and 3) post-hoc adaptation.\nJoint training.\nIn multi-task learning settings, modular task-specific components are trained jointly to mitigate catastrophic interference, with fixed or learned routing. Joint training can also provide a useful initialisation for modular parameters and allow for the additional of modular components in later stages.\nContinual learning.\nDuring continual learning, new modules are introduced into the model over time. The parameters of previous modules are typically frozen while new modules are connected to existing modules in different ways.\nPost-hoc adaptation.\nThese methods are also known as parameter-efficient fine-tuning as they are typically used to adapt a large pre-trained model to a target setting. We cover such methods for NLP in our\nEMNLP 2022 tutorial\n.\nPurposes of Modularity\nMany of the above methods are evaluated based on their ability to scale large models or enable few-shot transfer. Modularity is also crucial for other applications involving planning and systematic generalisation, including 1) hierarchical reinforcement learning; 2) neural programme induction; and 3) neural causal inference. We illustrate them below.\nDifferent purposes of modular deep learning.\n(a) Hierarchical Reinforcement Learning:\nPolicy sketches\nconsist of high-level policies (task instructions) that select low-level policies (options), which determine the choice of actions.\n(b) Programme Induction\n: The\nDifferentiable Neural Computer\nuses a recurrent neutral controller, which iteratively receives an input from the environment, writes to and reads from memory, and produces an output.\n(c) Causal Inference:\nCausal Independent Mechanisms\nroute a transformed example to an expert, which maps it to the original distribution. An adversarial discriminator attempts to distinguish between reconstructed and original examples.\nHierarchical reinforcement learning.\nIn order to learn over large time spans or with very sparse and delayed rewards in RL, it is often useful to learn intermediate abstractions, known as\noptions\nor\nskills\n, in the form of transferable sub-policies. Learning sub-policies introduces challenges related to specialisation and supervision and the space of actions and options. Strategies used to address them involve intrinsic rewards, sub-goals, and language as an intermediate space.\nProgramme simulation.\nModularity can also be used to simulate programmes by constructing a programme graph\ndynamically\nbased on an input or\nglobally\nbased on a task description. In addition to routing and computation functions, such architectures can be extended with an external memory. Programme simulation is useful when tasks rely on performing the correct sequence of sub-tasks.\nCausal inference.\nModularity in causal inference methods reflects the modularity in the (physical) mechanisms of the world. As modules are assumed to be independent and reusable, ML models mirroring this structure are more robust to interventions and local distribution shifts. Challenges include specialising each module towards a specific mechanism as well as jointly learning abstract representations and their interaction in a causal graph.\nApplications in Transfer Learning\nThe presented methods are used in a variety of applications. We first highlight common applications in NLP and then draw analogies to applications in speech, computer vision, and other areas of machine learning.\nMachine translation.\nIn MT,\nbilingual\nadapters have been used to adapt a massively multilingual NMT model to a specific source–target translation direction. Such work has been extended to more efficient\nmonolingual\nadapters. Hypernetworks have been used to enable positive transfer between languages. Other approaches such as language and domain-specific subnetworks and mixture-of-experts have also been applied.\nCross-lingual transfer.\nLanguage modules are combined with task modules to enable transfer of large models fine-tuned on a task in a source language to a different target language. Within this framework, many variations have been proposed that learn adapters for language pairs or language families, learn language and task subnetworks, or use a hypernetwork for the generation of various components.\nDomain adaptation.\nDomain-specific modular representations have been learned using adapters or subnetworks. A common design is to employ a set of shared modules and domain modules that are learned jointly, with additional regularisation or loss terms on the module parameters.\nKnowledge injection.\nModules can also be used to store and inject external knowledge, which can be combined with language, domain, or task knowledge. A common strategy is to train modules on synthetic data created based on the information in a knowledge base.\nSpeech processing.\nIn speech, similar methods have been explored as in NLP. The main differences are that the underlying model is typically a wav2vec variant and modular representations are optimised based on the CTC objective. The most common setting is to learn adapters for ASR.\nComputer vision and cross-modal learning.\nIn computer vision, common module choices are adapters and subnetworks based on ResNet or Vision Transformer models. For multi-modal learning, task and modality information are captured in separate modules for different applications. The recent\nFlamingo\nmodel, for instance, uses frozen pretrained vision and language models and learns new adapter layers to condition the language representations on visual inputs.\nFuture Directions\nFuture directions include combining different computation functions, gaining a better understanding of the nature and differences of different modular representations, integrating learned routing in pre-training, benchmarking routing methods, composing subnetwork information directly, developing learned aggregation methods, and creating extensible modular multi-task models, among others.\nConclusion\nWe have presented a categorisation of modularity in deep learning across four core dimensions. Given the trend of pre-training larger and larger models, we believe modularity will be crucial. It will enable more sustainable model development by modularising parts and developing modular methods that address current limitations and can be shared across different architectures. But modularity may also facilitate a shift away from a concentration of model development in a few institutions and to distributing the development of modular components across the community.\nCitation\nFor attribution in academic contexts or books, please cite\nour survey\nas:\nJonas Pfeiffer and Sebastian Ruder and Ivan Vulić and Edoardo M. Ponti, \"Modular Deep Learning\". arXiv preprint arXiv:2302.11529, 2023.\nBibTeX citation\n@article{pfeiffer2023modulardeeplearning,\n    author = {Pfeiffer, Jonas and Ruder, Sebastian and Vuli{'c}, Ivan and Ponti, Edoardo M.},\n    title = {{Modular Deep Learning}},\n    year = {2023},\n    journal = {CoRR},\n    volume = {abs/2302.11529},\n    url = {https://arxiv.org/abs/2302.11529},\n}\nPlease cite\nour tutorial\nas:\nSebastian Ruder and Jonas Pfeiffer and Ivan Vulić. \"Modular and Parameter-Efficient Fine-Tuning for {NLP} Models\". Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, 2022.\nBibTeX citation:\n@inproceedings{ruder-etal-2022-modular,\n    title = \"Modular and Parameter-Efficient Fine-Tuning for {NLP} Models\",\n    author = \"Ruder, Sebastian  and\n      Pfeiffer, Jonas  and\n      Vuli{\\'c}, Ivan\",\n    booktitle = \"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts\",\n    month = dec,\n    year = \"2022\",\n    address = \"Abu Dubai, UAE\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2022.emnlp-tutorials.5\",\n    pages = \"23--29\",\n}",
  "metadata": {
    "title": "Modular Deep Learning",
    "description": "An overview of modular deep learning across four dimensions (computation function, routing function, aggregation function, and training setting).",
    "keywords": null,
    "author": null,
    "og:site_name": "ruder.io",
    "og:type": "article",
    "og:title": "Modular Deep Learning",
    "og:description": "An overview of modular deep learning across four dimensions (computation function, routing function, aggregation function, and training setting).",
    "og:url": "https://www.ruder.io/modular-deep-learning/",
    "og:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2023/02/modular_case_studies_feature.png",
    "og:image:width": "1200",
    "og:image:height": "618",
    "twitter:card": "summary_large_image",
    "twitter:title": "Modular Deep Learning",
    "twitter:description": "An overview of modular deep learning across four dimensions (computation function, routing function, aggregation function, and training setting).",
    "twitter:url": "https://www.ruder.io/modular-deep-learning/",
    "twitter:image": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w1200/2023/02/modular_case_studies_feature.png",
    "twitter:label1": "Written by",
    "twitter:data1": "Sebastian Ruder",
    "twitter:site": "@seb_ruder",
    "article:published_time": "2023-02-23T09:07:51.000Z",
    "article:modified_time": "2024-10-23T13:20:52.000Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/7e30313ecf1c/code/body-000.txt",
      "chars": 136,
      "language": "text"
    },
    {
      "file": "attachments/7e30313ecf1c/code/body-001.txt",
      "chars": 294,
      "language": "text"
    },
    {
      "file": "attachments/7e30313ecf1c/code/body-002.txt",
      "chars": 229,
      "language": "text"
    },
    {
      "file": "attachments/7e30313ecf1c/code/body-003.txt",
      "chars": 542,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2023/02/modular_case_studies_feature.png",
      "bytes": 231093,
      "reused": true,
      "local_path": "attachments/7e30313ecf1c/images/001.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/size/w2000/2023/02/modular_case_studies_feature.png",
      "alt": "Modular Deep Learning",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2023/02/modular_case_studies.png",
      "bytes": 226288,
      "reused": true,
      "local_path": "attachments/7e30313ecf1c/images/005.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2023/02/modular_case_studies.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2023/02/computation_functions.png",
      "bytes": 151162,
      "reused": true,
      "local_path": "attachments/7e30313ecf1c/images/009.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2023/02/computation_functions.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2023/02/function_composition.png",
      "bytes": 222956,
      "reused": true,
      "local_path": "attachments/7e30313ecf1c/images/013.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2023/02/function_composition.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2023/02/routing_methods-2.png",
      "bytes": 119372,
      "reused": true,
      "local_path": "attachments/7e30313ecf1c/images/017.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2023/02/routing_methods-2.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2023/02/routing_levels.png",
      "bytes": 140238,
      "reused": true,
      "local_path": "attachments/7e30313ecf1c/images/021.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2023/02/routing_levels.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2023/02/purposes_of_modularity.png",
      "bytes": 295907,
      "reused": true,
      "local_path": "attachments/7e30313ecf1c/images/025.png",
      "src": "https://storage.ghost.io/c/bd/89/bd89d480-b8e0-47e6-8bae-dcb0d061b8d7/content/images/2023/02/purposes_of_modularity.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.ruder.io/author/sebastian/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.emnlp-main.617/",
      "label": "MAD-X",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2202.13914",
      "label": "Polytropon",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2101.03961",
      "label": "Mixture-of-Experts Transformer",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2302.11529?ref=ruder.io",
      "label": "our survey",
      "type": "a"
    },
    {
      "href": "https://tinyurl.com/modular-fine-tuning-tutorial?ref=ruder.io",
      "label": "EMNLP 2022 tutorial",
      "type": "a"
    },
    {
      "href": "https://www.modulardeeplearning.com/?ref=ruder.io",
      "label": "modulardeeplearning.com",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2020.emnlp-main.617/?ref=ruder.io",
      "label": "MAD-X",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2202.13914?ref=ruder.io",
      "label": "Polytropon",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2101.03961?ref=ruder.io",
      "label": "MoE Transformers",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2021.acl-long.353/?ref=ruder.io",
      "label": "Prefix tuning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1803.03635?ref=ruder.io",
      "label": "lottery ticket hypothesis",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v97/houlsby19a/houlsby19a.pdf?ref=ruder.io",
      "label": "Sequential Bottleneck Adapter",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v97/stickland19a/stickland19a.pdf?ref=ruder.io",
      "label": "Parallel Bottleneck Adapter",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2205.05638?ref=ruder.io",
      "label": "(IA)$^3$",
      "type": "a"
    },
    {
      "href": "https://docs.google.com/presentation/d/1seHOJ7B0bQEPJ3LBW5VmruMCILiVRoPb8nmU2OS-Eqc/edit?ref=ruder.io",
      "label": "tutorial slides",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1611.01796?ref=ruder.io",
      "label": "Policy sketches",
      "type": "a"
    },
    {
      "href": "https://www.nature.com/articles/nature20101?ref=ruder.io",
      "label": "Differentiable Neural Computer",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1712.00961?ref=ruder.io",
      "label": "Causal Independent Mechanisms",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2204.14198?ref=ruder.io",
      "label": "Flamingo",
      "type": "a"
    },
    {
      "href": "https://www.ruder.io/modular-deep-learning/tinyurl.com/modular-fine-tuning-tutorial",
      "label": "our tutorial",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:56Z",
  "source_url": "https://www.ruder.io/modular-deep-learning/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2202.13914",
      "bytes": 800125,
      "reused": false,
      "local_path": "papers/repair/ba4356d00186b275b410.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2101.03961",
      "bytes": 1304157,
      "reused": true,
      "local_path": "papers/repair/a780b9df79db7592bd38.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2302.11529",
      "bytes": 8145325,
      "reused": false,
      "local_path": "papers/repair/dd3038a0c4c64c5319dc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2202.13914",
      "bytes": 800125,
      "reused": true,
      "local_path": "papers/repair/ba4356d00186b275b410.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2101.03961",
      "bytes": 1304157,
      "reused": true,
      "local_path": "papers/repair/a780b9df79db7592bd38.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1803.03635",
      "bytes": 4001475,
      "reused": true,
      "local_path": "papers/repair/85eb00976bcf06c5e750.pdf"
    },
    {
      "status": "ok",
      "url": "http://proceedings.mlr.press/v97/houlsby19a/houlsby19a.pdf?ref=ruder.io",
      "bytes": 724447,
      "reused": false,
      "local_path": "papers/repair/d30e94c6755c97464157.pdf"
    },
    {
      "status": "ok",
      "url": "http://proceedings.mlr.press/v97/stickland19a/stickland19a.pdf?ref=ruder.io",
      "bytes": 257844,
      "reused": false,
      "local_path": "papers/repair/81b427b0a9903b40be30.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2205.05638",
      "bytes": 562785,
      "reused": false,
      "local_path": "papers/repair/e37211f681267431727a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1611.01796",
      "bytes": 1457886,
      "reused": false,
      "local_path": "papers/repair/a23d4b63c63a52786ff9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1712.00961",
      "bytes": 1124132,
      "reused": false,
      "local_path": "papers/repair/b7573af8a16064507b6b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2204.14198",
      "bytes": 31579390,
      "reused": true,
      "local_path": "papers/repair/0957ce92a2e9701d5e26.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 19879,
  "expected_body_images": 7,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/7e30313ecf1c/pages/source.html",
  "local_html": "attachments/7e30313ecf1c/pages/article.html",
  "body_base_path": "attachments/7e30313ecf1c/pages/",
  "body_markdown_path": "attachments/7e30313ecf1c/pages/article.md"
}