{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Maxime Labonne",
  "title_listed": "Create Mixtures of Experts with MergeKit",
  "keywords": "MoE,模型合并,MergeKit",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "部分免费/会员墙",
  "url": "https://towardsdatascience.com/create-mixtures-of-experts-with-mergekit-11b318c99562/",
  "publish_date": "2024-03",
  "corpus_stage": "预训练/SFT",
  "notes": "",
  "id": "a98367285fab",
  "collected_at": "2026-10-01T21:50:39",
  "title": "Create Mixtures of Experts with MergeKit | Towards Data Science",
  "body_markdown": "Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# Create Mixtures of Experts with MergeKit\nCombine multiple models into a single MoE\n[Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\nMarch 27, 20249 min read\n### _Combine multiple models into a single MoE_\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/1kJ4ALMDzNaq9W88ZtVnhiQ.jpeg)\nImage by author\nThanks to the release of Mixtral, the **Mixture of Experts** (MoE) architecture has become popular in recent months. This architecture offers an interesting tradeoff: higher performance at the cost of increased VRAM usage. While Mixtral and other MoE architectures are pre-trained from scratch, another method of creating MoE has recently appeared. Thanks to Arcee's [MergeKit](https://github.com/arcee-ai/mergekit) library, we now have a new way of creating MoEs by ensembling several pre-trained models. These are often referred to as **frankenMoEs** or **MoErges** to distinguish them from the pre-trained MoEs.\nIn this article, we will detail how the MoE architecture works and how frankenMoEs are created. Finally, we will make our [own frankenMoE](https://huggingface.co/mlabonne/Beyonder-4x7B-v3) with MergeKit and evaluate it on several benchmarks. The code is available on Google Colab in a wrapper called [LazyMergeKit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb#scrollTo=d5mYzDo1q96y).\nSpecial thanks to [Charles Goddard](https://github.com/cg123), the creator of MergeKit, for proofreading this article.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer), [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) and [Computer Science](https://roadmap.sh/computer-science) roadmaps.\n## 🔀 Introduction to MoEs\nA Mixture of Experts is an architecture designed for improved efficiency and performance. It uses multiple specialized subnetworks, known as \"**experts**.\" Unlike dense models, where the entire network is activated, MoEs only activate relevant experts based on the input. This results in faster training and more efficient inference.\nThere are two components at the core of an MoE model:\n  1. **Sparse MoE Layers** : These replace the dense feed-forward network layers in the transformer architecture. Each MoE layer contains several experts, and only a subset of these experts are engaged for a given input.\n  2. **Gate Network or Router** : This component determines which tokens are processed by which experts, ensuring that each part of the input is handled by the most suitable expert(s).\n\n\nIn the following example, we show how a Mistral-7B block is transformed into an MoE block with a sparse MoE layer (feedforward network 1, 2, and 3) and a router. This example represents an MoE with three experts, where two are currently engaged (FFN 1 and FFN 3).\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/1gGQxuStcGfSS_Boy9rKvAw.png)\nImage by author\nMoEs also come with their own set of challenges, especially in terms of fine-tuning and memory requirements. The fine-tuning process can be difficult due to the model's complexity, with the need to **balance expert usage** during training to properly train the gating weights to select the most relevant ones. In terms of memory, even though only a fraction of the total parameters are used during inference, the entire model, including all experts, needs to be **loaded into memory** , which requires high VRAM capacity.\nMore specifically, there are two essential parameters when it comes to MoEs:\n  * **Number of experts** (`num_local_experts`): This determines the total number of experts in the architecture (e.g., 8 for Mixtral). The higher the number of experts, the higher the VRAM usage.\n  * **Number of experts/token** (`num_experts_per_tok`): This determines the number of experts that are engaged for each token and each layer (e.g., 2 for Mixtral). There is a tradeoff between a high number of experts per token for accuracy (but diminishing returns) vs. a low number for fast training and inference.\n\n\nHistorically, MoEs have underperformed dense models. However, the release of [Mixtral-8x7B](https://arxiv.org/abs/2401.04088) in December 2023 shook things up and showed impressive performance for its size. Additionally, GPT-4 is also rumored to be an MoE, which would make sense as it would be a lot cheaper to run and train for OpenAI compared to a dense model. In addition to these recent excellent MoEs, we now have a new way of creating MoEs with MergeKit: frankenMoEs, also called MoErges.\n## 🧟‍♂️ True MoEs vs. frankenMoEs\nThe main difference between true MoEs and frankenMoEs is how they're trained. In the case of true MoEs, the experts and the router are trained jointly. In the case of frankenMoEs, we upcycle existing models and initialize the router afterward.\nIn other words, we copy the weights of the layer norm and self-attention layers from a base model, and then copy the weights of the FFN layers found in each expert. This means that besides the FFNs, all the other parameters are shared. This explains why Mixtral-8x7B with eight experts doesn't have 8*7 = 56B parameters, but about 45B. This is also why using two experts per token gives the inference speed (FLOPs) of a 12B dense model instead of 14B.\nFrankenMoEs are about selecting the most relevant experts and initializing them properly. MergeKit currently implements three ways of initializing the routers:\n  1. [**Random**](https://github.com/arcee-ai/mergekit/blob/9c691527f7192b5a2fc388555bfd3105e0898480/mergekit/scripts/mixtral_moe.py#L139-L142): Random weights. Be careful when using it as the same experts might be selected every time (it requires further fine-tuning or `num_local_experts = num_experts_per_tok`, which means you don't need any routing).\n  2. [**Cheap embed**](https://github.com/arcee-ai/mergekit/blob/9c691527f7192b5a2fc388555bfd3105e0898480/mergekit/scripts/mixtral_moe.py#L91C1-L109C37): It uses the raw embeddings of the input tokens directly and applies the same transformation across all layers. This method is computationally inexpensive and suitable for execution on less powerful hardware.\n  3. [**Hidden**](https://github.com/arcee-ai/mergekit/blob/9c691527f7192b5a2fc388555bfd3105e0898480/mergekit/scripts/mixtral_moe.py#L70-L88): It creates hidden representations of a list of positive and negative prompts by extracting them from the last layer of the LLM. They are averaged and normalized to initialize the gates. More information about it is available on [Charles Goddard's blog](https://goddard.blog/posts/clown-moe/).\n\n\nAs you can guess, the \"hidden\" initialization is the most efficient to correctly route the tokens to the most relevant experts. In the next section, we will create our own frankenMoE using this technique.\n## 💻 Creating a frankenMoE\nTo create our frankenMoE, we need to select `n` experts. In this case, we will rely on Mistral-7B thanks to its popularity and relatively small size. However, eight experts like in Mixtral is quite a lot, as we need to fit all of them in memory. For efficiency, I'll only use four experts in this example, with two of them engaged for each token and each layer. In this case, we will end up with a model with 24.2B parameters instead of 4*7 = 28B parameters.\nHere, our goal is to create a well-rounded model that can do pretty much everything: write stories, explain articles, code in Python, etc. We can decompose this requirement into four tasks and select the best expert for each of them. This is how I decomposed it:\n  * **Chat model** : a general-purpose model that is used in most interactions. I used [mlabonne/AlphaMonarch-7B](https://huggingface.co/mlabonne/AlphaMonarch-7B), which perfectly satisfies the requirements.\n  * **Code model** : a model capable of generating good code. I don't have a lot of experience with Mistral-7B-based code models, but I found [beowolx/CodeNinja-1.0-OpenChat-7B](https://huggingface.co/beowolx/CodeNinja-1.0-OpenChat-7B) particularly good compared to others.\n  * **Math model** : math is tricky for LLMs, which is why we want a model specialized in math. Thanks to its high MMLU and GMS8K scores, I chose [mlabonne/NeuralDaredevil-7B](https://huggingface.co/mlabonne/NeuralDaredevil-7B) for this purpose.\n  * **Role-play model** : The goal of this model is to write high-quality stories and conversations. I selected [SanjiWatsuki/Kunoichi-DPO-v2–7B](https://huggingface.co/SanjiWatsuki/Kunoichi-DPO-v2-7B) because of its good reputation and high MT-Bench score (8.51 vs. 8.30 for Mixtral).\n\n\nNow that we've identified the experts we want to use, we can create the YAML configuration that MergeKit will use to create our frankenMoE. This uses the mixtral branch of MergeKit. You can find more information about how to write the configuration [on this page](https://github.com/arcee-ai/mergekit/blob/mixtral/docs/moe.md). Here is our version:\nmakefile\n\n```\nbase_model: mlabonne/AlphaMonarch-7B\nexperts:\n  - source_model: mlabonne/AlphaMonarch-7B\n    positive_prompts:\n    - \"chat\"\n    - \"assistant\"\n    - \"tell me\"\n    - \"explain\"\n    - \"I want\"\n  - source_model: beowolx/CodeNinja-1.0-OpenChat-7B\n    positive_prompts:\n    - \"code\"\n    - \"python\"\n    - \"javascript\"\n    - \"programming\"\n    - \"algorithm\"\n  - source_model: SanjiWatsuki/Kunoichi-DPO-v2-7B\n    positive_prompts:\n    - \"storywriting\"\n    - \"write\"\n    - \"scene\"\n    - \"story\"\n    - \"character\"\n  - source_model: mlabonne/NeuralDaredevil-7B\n    positive_prompts:\n    - \"reason\"\n    - \"math\"\n    - \"mathematics\"\n    - \"solve\"\n    - \"count\"\n```\n\nFor each expert, I provide five basic positive prompts. You can be a bit fancier and write entire sentences if you want. The best strategy consists of using real prompts that should trigger a particular expert. You can also add negative prompts to do the opposite.\nOnce this is ready, you can save your configuration as `config.yaml`. In the same folder, we will download and install the [mergekit](https://github.com/arcee-ai/mergekit) library (mixtral branch).\nbash\n\n```\ngit clone -b mixtral https://github.com/arcee-ai/mergekit.git\ncd mergekit && pip install -e .\npip install -U transformers\n```\n\nIf your computer has enough RAM (roughly 24–32 GB of RAM), you can run the following command:\nlua\n\n```\nmergekit-moe config.yaml merge --copy-tokenizer\n```\n\nIf you don't have enough RAM, you can shard the models instead as follows (it will take longer):\ncss\n\n```\nmergekit-moe config.yaml merge --copy-tokenizer --allow-crimes --out-shard-size 1B --lazy-unpickle\n```\n\nThis command automatically downloads the experts and creates the frankenMoE in the `merge` directory. For the `hidden` gate mode, you can also use the `--load-in-4bit` and `--load-in-8bit` options to compute hidden states with lower precision.\nAlternatively, you can copy your configuration into [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb#scrollTo=d5mYzDo1q96y), a wrapper I made to simplify model merging. In this Colab notebook, you can input your model name, select the `mixtral` branch, specify your Hugging Face username/token, and run the cells. After creating your frankenMoE, it will also upload it to the Hugging Face Hub with a nicely formatted model card.\nI called my model [Beyonder-4x7B-v3](https://huggingface.co/mlabonne/Beyonder-4x7B-v3) and created [GGUF versions](https://huggingface.co/mlabonne/Beyonder-4x7B-v3-GGUF) of it using [AutoGGUF](https://colab.research.google.com/drive/1P646NEg33BZy4BfLDNpTz0V0lwIU3CHu#scrollTo=fD24jJxq7t3k). If you can't run GGUF versions on your local machine, you can also perform inference using this [Colab notebook](https://colab.research.google.com/drive/1SIfwhpLttmoZxT604LGVXDOI9UKZ_1Aq?usp=sharing).\nTo get a good overview of its capabilities, it has been evaluated on three different benchmarks: Nous' benchmark suite, EQ-Bench, and the Open LLM Leaderboard. This model is not designed to excel in traditional benchmarks, as the code and role-playing models generally do not apply to those contexts. Nonetheless, it performs remarkably well thanks to strong general-purpose experts.\n**Nous** : Beyonder-4x7B-v3 is one of the best models on Nous' benchmark suite (evaluation performed using [LLM AutoEval](https://github.com/mlabonne/llm-autoeval)) and significantly outperforms the v2. See the entire leaderboard [here](https://huggingface.co/spaces/mlabonne/Yet_Another_LLM_Leaderboard).\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/0MTWS0BgqX17yq9qa.png)\n**EQ-Bench** : It's also the best 4x7B model on the [EQ-Bench leaderboard](https://eqbench.com/), outperforming older versions of ChatGPT and Llama-2–70b-chat. Beyonder is very close to Mixtral-8x7B-Instruct-v0.1 and Gemini Pro, which are (supposedly) much bigger models.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/0ebG79Y254SJp58KG.png)\n**Open LLM Leaderboard** : Finally, it's also a strong performer on the Open LLM Leaderboard, significantly outperforming the v2 model.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/0RpN_fYTFfzxvSJ66.png)\nOn top of these quantitative evaluations, I recommend checking the model's outputs in a more qualitative way using a GGUF version on [LM Studio](https://lmstudio.ai/). A common way of testing these models is to gather a private set of questions and check their outputs. With this strategy, I found that Beyonder-4x7B-v3 is quite robust to changes in the user and system prompts compared to other models, including AlphaMonarch-7B. This is pretty cool as it improves the usefulness of the model in general.\nFrankenMoEs are a promising but still experimental approach. The trade-offs, like higher VRAM demand and slower inference speeds, can make it challenging to see their advantage over simpler merging techniques like SLERP or DARE TIES. Especially, when you use frankenMoEs with just two experts, they might not perform as well as if you had simply merged the two models. However, frankenMoEs excel in preserving knowledge, which can result in stronger models, as demonstrated by Beyonder-4x7B-v3. With the right hardware, these drawbacks can be effectively mitigated.\n## Conclusion\nIn this article, we introduced the Mixture of Experts architecture. Unlike traditional MoEs that are trained from scratch, MergeKit facilitates the creation of MoEs by ensembling experts, offering an innovative approach to improving model performance and efficiency. We detailed the process of creating a frankenMoE with MergeKit, highlighting the practical steps involved in selecting and combining different experts to produce a high-quality MoE.\nThanks for reading this article. I encourage you to try to make your own FrankenMoEs using [LazyMergeKit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb#scrollTo=d5mYzDo1q96y): select a few models, create your config based Beyonder's, and run the notebook to create your own models! If you liked this article, please follow me on [Hugging Face](https://huggingface.co/mlabonne) and X/Twitter [@maximelabonne](https://twitter.com/maximelabonne).\n## References\n  * [Mixtral of Experts](https://arxiv.org/abs/2401.04088) by Jiang et al. (2023)\n  * [Mixture of Experts for Clowns](https://goddard.blog/posts/clown-moe/) by Charles Goddard (2023)\n  * [Mixture of Experts Explained](https://huggingface.co/blog/moe) by Sanseviero et al. (2023)\n  * [Adaptive Mixture of Local Experts](https://www.cs.toronto.edu/~hinton/absps/jjnh91.pdf) by Jacobs et al. (1991)\n  * [Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints](https://arxiv.org/abs/2212.05055) by Komatsuzaki et al. (2022)\n\n\n* * *\n_Learn more about machine learning and support my work with one click - become a Medium member here:_\n> [**Join Medium with my referral link - Maxime Labonne**](https://medium.com/@mlabonne/membership)\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657499317_sxy3ci.png)\nWritten By\n## [Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[See all from Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Data Science](https://towardsdatascience.com/tag/data-science/)[Large Language Models](https://towardsdatascience.com/tag/large-language-models/)[Programming](https://towardsdatascience.com/tag/programming/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fcreate-mixtures-of-experts-with-mergekit-11b318c99562%2F&title=Create%20Mixtures%20of%20Experts%20with%20MergeKit)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fcreate-mixtures-of-experts-with-mergekit-11b318c99562%2F&title=Create%20Mixtures%20of%20Experts%20with%20MergeKit)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fcreate-mixtures-of-experts-with-mergekit-11b318c99562%2F&text=Create%20Mixtures%20of%20Experts%20with%20MergeKit)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/1tXpPB6eldPPHsZaWsMV9yw.png)](https://towardsdatascience.com/websites-for-statistics-and-data-analysis-on-every-device-ebf92bec3e53/)\n## [Websites for Statistics and Data Analysis on Every Device](https://towardsdatascience.com/websites-for-statistics-and-data-analysis-on-every-device-ebf92bec3e53/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nMy selection of online tools for data analysis in web browsers.\n[Luciano Abriata](https://towardsdatascience.com/author/lucianosphere/)\nOctober 19, 202111 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/06/1TrYnSLziZvr9AOjxXKIWwQ.png)](https://towardsdatascience.com/visualizing-bias-in-data-using-embedding-projector-649bc65e7487/)\n## [Visualizing Bias in Data using Embedding Projector](https://towardsdatascience.com/visualizing-bias-in-data-using-embedding-projector-649bc65e7487/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nUsing Open-Sourced Embedding Projector tool for Interactive Visualization and Interpretation of Embeddings\n[Parul Pandey](https://towardsdatascience.com/author/pandeyparul/)\nJune 6, 20197 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/06/1lovxQO12mwBmO3mhK2QeIg-scaled.jpeg)](https://towardsdatascience.com/serverless-model-hosting-with-docker-aws-lambda-and-api-gateway-2d79382ecb45/)\n## [Serverless Model Hosting with Docker, AWS Lambda and API Gateway](https://towardsdatascience.com/serverless-model-hosting-with-docker-aws-lambda-and-api-gateway-2d79382ecb45/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nMake the most of Docker support in Lambda to host your models without the need for dedicated servers.\n[Jonathan Readshaw](https://towardsdatascience.com/author/readshawjonathan/)\nJune 20, 20213 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/1wsR4Vh-MX3XUqajQnssXbQ.png)](https://towardsdatascience.com/the-augmentation-engineers-1ac2a676710a/)\n## [Augmentation is All You Need](https://towardsdatascience.com/the-augmentation-engineers-1ac2a676710a/)\n[](https://towardsdatascience.com/create-mixtures-of-experts-with-mergekit-11b318c99562/)\nHow an almost trivial training trick became the center of the recent AI shift towards self-supervised learning.\n[Jonathan Laserson, PhD](https://towardsdatascience.com/author/jonilaserson/)\nAugust 16, 202112 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/01/1bwqDfb8L8yF-n3O8ECKr1w-scaled.jpeg)](https://towardsdatascience.com/tradable-patterns-hiding-in-plain-sight-350cc58f0860/)\n## [Tradable Patterns Hiding in Plain Sight](https://towardsdatascience.com/tradable-patterns-hiding-in-plain-sight-350cc58f0860/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nThere are longer-term tradable patterns in the stock market that you don't need to be a professional trader or statistics PhD to figure...\n[Bryant Sheehy](https://towardsdatascience.com/author/bnsheehy/)\nJanuary 29, 202012 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1GpUjq8rjOJ6kU292RRo1IA.jpeg)](https://towardsdatascience.com/the-google-data-analytics-professional-certificate-a-reflection-5264982365f9/)\n## [The Google Data Analytics Professional Certificate: A Reflection](https://towardsdatascience.com/the-google-data-analytics-professional-certificate-a-reflection-5264982365f9/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nA solid foundation to build upon\n[J Edgar Mihelic](https://towardsdatascience.com/author/mihelic/)\nApril 19, 20215 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0hPVl2IGzZ3mQ4afc-scaled.jpg)](https://towardsdatascience.com/the-specialized-data-scientist-will-win-in-the-long-run-22c47342aa00/)\n## [The Specialized \"Data Scientist\" Will Win in The Long-Run](https://towardsdatascience.com/the-specialized-data-scientist-will-win-in-the-long-run-22c47342aa00/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nBecoming a Specialist Will Take You Further than A Generalist\n[Kurtis Pykes](https://towardsdatascience.com/author/kurtispykes/)\nOctober 14, 20204 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/PANet-Cover-Page.jpg)](https://towardsdatascience.com/panet-paper-walkthrough-when-feature-pyramids-go-bottom-up/)\n## [PANet Paper Walkthrough: When Feature Pyramids Go Bottom-Up](https://towardsdatascience.com/panet-paper-walkthrough-when-feature-pyramids-go-bottom-up/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nUnderstanding how PANet shortens the path between low-level and high-level features\n[Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nJuly 5, 202615 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/12/1g-UHw72Rq4grQRDmzAHp3g.png)](https://towardsdatascience.com/an-unbiased-guide-to-bias-in-ai-3841c2b36165/)\n## [An \"Unbiased\" Guide to Bias in AI](https://towardsdatascience.com/an-unbiased-guide-to-bias-in-ai-3841c2b36165/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nEthical vs. statistical bias in AI/ML models\n[Shahrokh Barati](https://towardsdatascience.com/author/tech.intrapreneur/)\nDecember 15, 202211 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\nword word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word\n  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1  \nmmMwWLliI0fiflO&1\n",
  "metadata": {
    "title": "Create Mixtures of Experts with MergeKit | Towards Data Science",
    "description": "Combine multiple models into a single MoE",
    "keywords": null,
    "author": "Maxime Labonne",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Create Mixtures of Experts with MergeKit",
    "og:description": "Combine multiple models into a single MoE",
    "og:url": "https://towardsdatascience.com/create-mixtures-of-experts-with-mergekit-11b318c99562/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/1kJ4ALMDzNaq9W88ZtVnhiQ.jpeg",
    "og:image:width": "1792",
    "og:image:height": "1024",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Maxime Labonne",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "9 min read",
    "article:published_time": "2024-03-27T13:47:23.000Z",
    "article:modified_time": "2024-03-27T13:47:23.000Z",
    "article:author": "https://towardsdatascience.com/author/mlabonne/"
  },
  "paywall": {
    "listed_paid": true,
    "detected": false,
    "marker": "",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": "",
    "retried": true
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/a98367285fab/code/block_000.txt",
      "chars": 645
    },
    {
      "language": "txt",
      "file": "attachments/a98367285fab/code/block_001.txt",
      "chars": 130
    },
    {
      "language": "txt",
      "file": "attachments/a98367285fab/code/block_002.txt",
      "chars": 48
    },
    {
      "language": "txt",
      "file": "attachments/a98367285fab/code/block_003.txt",
      "chars": 99
    }
  ],
  "images": [
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/1kJ4ALMDzNaq9W88ZtVnhiQ.jpeg",
      "alt": "Image by author",
      "local_path": "attachments/a98367285fab/images/000.jpeg",
      "bytes": 405035,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/03/1gGQxuStcGfSS_Boy9rKvAw.png",
      "alt": "Image by author",
      "local_path": "attachments/a98367285fab/images/001.png",
      "bytes": 137079,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/1tXpPB6eldPPHsZaWsMV9yw.png",
      "alt": "",
      "local_path": "attachments/a98367285fab/images/002.png",
      "bytes": 178639,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/06/1TrYnSLziZvr9AOjxXKIWwQ.png",
      "alt": "",
      "local_path": "attachments/a98367285fab/images/003.png",
      "bytes": 237562,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/06/1lovxQO12mwBmO3mhK2QeIg-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/a98367285fab/images/004.jpeg",
      "bytes": 274138,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/1wsR4Vh-MX3XUqajQnssXbQ.png",
      "alt": "",
      "local_path": "attachments/a98367285fab/images/005.png",
      "bytes": 529694,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/01/1bwqDfb8L8yF-n3O8ECKr1w-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/a98367285fab/images/006.jpeg",
      "bytes": 549054,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1GpUjq8rjOJ6kU292RRo1IA.jpeg",
      "alt": "",
      "local_path": "attachments/a98367285fab/images/007.jpeg",
      "bytes": 34757,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0hPVl2IGzZ3mQ4afc-scaled.jpg",
      "alt": "",
      "local_path": "attachments/a98367285fab/images/008.jpg",
      "bytes": 300933,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/PANet-Cover-Page.jpg",
      "alt": "",
      "local_path": "attachments/a98367285fab/images/009.jpg",
      "bytes": 158881,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/12/1g-UHw72Rq4grQRDmzAHp3g.png",
      "alt": "",
      "local_path": "attachments/a98367285fab/images/010.png",
      "bytes": 156371,
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [],
  "retry_at": "2026-10-03T07:59:07",
  "repair_version": 2,
  "repaired_at": "2026-10-14T11:11:48Z",
  "source_url": "https://towardsdatascience.com/create-mixtures-of-experts-with-mergekit-11b318c99562/",
  "delivery_status": "excluded_paid",
  "discovery_status": "excluded_paid",
  "missing_reason": "按原约定排除付费/会员全文"
}