{ "source_file": "0930", "source_platform": "Medium", "author": "Maxime Labonne", "title_listed": "Merge Large Language Models with mergekit", "keywords": "模型合并,mergekit", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "部分免费/会员墙", "url": "https://towardsdatascience.com/merge-large-language-models-with-mergekit-2118fb392b54/", "publish_date": "2024-01", "corpus_stage": "预训练/SFT", "notes": "", "id": "3409cf6aecf7", "collected_at": "2026-10-01T21:53:34", "title": "Merge Large Language Models with mergekit | Towards Data Science", "body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# Merge Large Language Models with mergekit\nCreate your own models easily, no GPU required!\n[Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\nJanuary 8, 202411 min read\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/1kBMm6Dgfej-snEAC7dXDjg.jpeg)\nImage by author\nModel merging is a technique that **combines two or more LLMs** into a single model. It's a relatively new and experimental method to create new models for cheap (no GPU required). Model merging works surprisingly well and produced many state-of-the-art models on the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).\nIn this tutorial, we will implement it using the [mergekit](https://github.com/cg123/mergekit) library. More specifically, we will review four merge methods and provide examples of configurations. Then, we will use mergekit to create our own model, [Marcoro14–7B-slerp](https://huggingface.co/mlabonne/Marcoro14-7B-slerp), which became the best-performing model on the Open LLM Leaderboard (02/01/24).\nThe code is available on [GitHub](https://github.com/mlabonne/llm-course/blob/main/Mergekit.ipynb) and [Google Colab](https://colab.research.google.com/drive/1_JS7JKJAQozD48-LhYdegcuuZ2ddgXfr?usp=sharing). I recommend using my automated notebook to easily run mergekit: [🥱 LazyMergekit.](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing)\n_A special thanks to_[ _Charles Goddard_](https://www.linkedin.com/in/charles-goddard-7b6797b/) _, the author of the mergekit library, for reviewing this article._\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer), [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) and [Computer Science](https://roadmap.sh/computer-science) roadmaps.\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/1Gm1kcCkX76hMD7dMK1AJ7Q.png)\nImage by author\n## 🤝 Merge algorithms\nIn this section, we will focus on four methods currently implemented in [mergekit](https://github.com/cg123/mergekit). Note that there are other methods, such as [linear](https://github.com/cg123/mergekit/tree/1011ef3a84e4c5545473602baf7ef32d535044a9#linear) and [Task Arithmetic](https://arxiv.org/abs/2212.04089). If you're interested in papers on model merging, I recommend [this excellent collection](https://huggingface.co/collections/osanseviero/model-merging-65097893623330a3a51ead66) on Hugging Face.\n### 1. SLERP\n**Spherical Linear Interpolation** (SLERP) is a method used to smoothly interpolate between two vectors. It maintains a constant rate of change and preserves the geometric properties of the spherical space in which the vectors reside.\nThere are several reasons to prefer SLERP over a traditional linear interpolation. For example, in high-dimensional spaces, linear interpolation can lead to a **decrease in the magnitude** of the interpolated vector (i.e., it reduces the scale of weights). Moreover, the change in direction of the weights often represents **more meaningful information** (like feature learning and representation) than the magnitude of change.\nSLERP is implemented using the following steps:\n 1. Normalize the input vectors to unit length, ensuring they represent directions rather than magnitudes\n 2. Calculate the angle between these vectors using their dot product.\n 3. If the vectors are nearly collinear, it defaults to linear interpolation for efficiency. Otherwise, SLERP computing scale factors based on the interpolation factor `t` (`t=0` = 100% of the first vector, `t=1` = 100% of model 2) and the angle between the vectors.\n 4. These factors are used to weigh the original vectors, which are then summed to obtain the interpolated vector.\n\n\nSLERP is currently the most popular merging method, but it is limited to combining only two models at a time. It is still possible to hierarchically combine multiple models, as shown in [Mistral-7B-Merge-14-v0.1](https://huggingface.co/EmbeddedLLM/Mistral-7B-Merge-14-v0.1).\n_Example of configuration:_\nyaml\n\n```\nslices:\n - sources:\n - model: OpenPipe/mistral-ft-optimized-1218\n layer_range: [0, 32]\n - model: mlabonne/NeuralHermes-2.5-Mistral-7B\n layer_range: [0, 32]\nmerge_method: slerp\nbase_model: OpenPipe/mistral-ft-optimized-1218\nparameters:\n t:\n - filter: self_attn\n value: [0, 0.5, 0.3, 0.7, 1]\n - filter: mlp\n value: [1, 0.5, 0.7, 0.3, 0]\n - value: 0.5\ndtype: bfloat16\n```\n\nThis is a classic SLERP configuration, applied to every layer of both models. Note that we input a gradient of values for the interpolation factor `t`. The parameters for the self-attention and MLP layers will use different combinations of [OpenPipe/mistral-ft-optimized-1218](https://huggingface.co/OpenPipe/mistral-ft-optimized-1218) and [mlabonne/NeuralHermes-2.5-Mistral-7B](https://huggingface.co/mlabonne/NeuralHermes-2.5-Mistral-7B). The other layers are a 50/50 mixture of the two models.\nYou can find the final model on the Hugging Face Hub at [mlabonne/NeuralPipe-7B-slerp](https://huggingface.co/mlabonne/NeuralPipe-7B-slerp).\n### 2. TIES\nIntroduced in [this paper](https://arxiv.org/abs/2306.01708) by Yadav et al., **TIES-Merging** is designed to efficiently merge multiple task-specific models into a single multitask model. It addresses two main challenges in model merging:\n * **Redundancy in model parameters** : It identifies and eliminates redundant parameters within task-specific models. This is achieved by focusing on the changes made during fine-tuning, identifying the top-k% most significant changes, and discarding the rest.\n * **Disagreement between parameter signs** : Conflicts arise when different models suggest opposing adjustments to the same parameter. TIES-Merging resolves these conflicts by creating a unified sign vector that represents the most dominant direction of change across all models.\n\n\nTIES-Merging is divided into the following three steps:\n 1. **Trim** : Reduces redundancy in task-specific models by retaining only a fraction the most significant parameters (density parameter) and resetting the rest to zero.\n 2. **Elect Sign** : Resolves sign conflicts across different models by creating a unified sign vector based on the most dominant direction (positive or negative) in terms of cumulative magnitude.\n 3. **Disjoint Merge** : Averages parameter values that align with the unified sign vector, excluding zero values.\n\n\nUnlike SLERP, TIES can merge multiple models at a time.\n_Example of configuration:_\nyaml\n\n```\nmodels:\n - model: mistralai/Mistral-7B-v0.1\n # no parameters necessary for base model\n - model: OpenPipe/mistral-ft-optimized-1218\n parameters:\n density: 0.5\n weight: 0.5\n - model: mlabonne/NeuralHermes-2.5-Mistral-7B\n parameters:\n density: 0.5\n weight: 0.3\nmerge_method: ties\nbase_model: mistralai/Mistral-7B-v0.1\nparameters:\n normalize: true\ndtype: float16\n```\n\nWith this config, we use Mistral-7B as a base model to calculate the delta weights. We merge the same two models: [mistral-ft-optimized-1218](https://huggingface.co/OpenPipe/mistral-ft-optimized-1218) (50%) and [NeuralHermes-2.5-Mistral-7B](https://huggingface.co/mlabonne/NeuralHermes-2.5-Mistral-7B) (30%) with normalization. Here, the density means that we're only retaining 50% of the parameters of each model (the other half comes from the base model).\nNote that the sum of the weights is not equal to 1 in the config, but the `normalize: true` parameter will automatically normalize them internally. This config is inspired by the parameters provided by the author of [OpenHermes-2.5-neural-chat-7b-v3–1–7B](https://huggingface.co/Weyaxi/OpenHermes-2.5-neural-chat-7b-v3-1-7B).\nYou can find the final model on the Hugging Face Hub at [mlabonne/NeuralPipe-7B-ties](https://huggingface.co/mlabonne/NeuralPipe-7B-ties).\n### 3. DARE\nIntroduced by Yu et al. (2023), [DARE](https://arxiv.org/abs/2311.03099) uses an approach similar to TIES with two main differences:\n * **Pruning** : DARE randomly reset fine-tuned weights to their original values (those of the base model).\n * **Rescaling** : DARE rescales the weights to keep the expectations of model outputs approximately unchanged. It adds the rescaled weights of both (or more) models to the weights of the base model with a scale factor.\n\n\nMergekit's implementation of this method has two flavors: with the sign election step of TIES (`dare_ties`) or without (`dare_linear`).\n_Example of configuration:_\nyaml\n\n```\nmodels:\n - model: mistralai/Mistral-7B-v0.1\n # No parameters necessary for base model\n - model: samir-fama/SamirGPT-v1\n parameters:\n density: 0.53\n weight: 0.4\n - model: abacusai/Slerp-CM-mist-dpo\n parameters:\n density: 0.53\n weight: 0.3\n - model: EmbeddedLLM/Mistral-7B-Merge-14-v0.2\n parameters:\n density: 0.53\n weight: 0.3\nmerge_method: dare_ties\nbase_model: mistralai/Mistral-7B-v0.1\nparameters:\n int8_mask: true\ndtype: bfloat16\n```\n\nIn this configuration, we merge three different models based on Mistral-7B using `dare_ties`. This time, I chose weights that sum to 1 (the sum should be between 0.9 and 1.1). The density parameter is a little higher than what's recommended in the paper (<0.5), but it looks like it gives consistently better results (see [this discussion](https://github.com/cg123/mergekit/issues/26)).\nYou can find it on the Hugging Face Hub at [mlabonne/Daredevil-7B](https://huggingface.co/mlabonne/Daredevil-7B). It's also the best merge model in this article, outperforming even Marcoro14–7B-slerp.\n### 4. Passthrough\nThe passthrough method differs significantly from the previous ones. By concatenating layers from different LLMs, it can produce models with an **exotic number of parameters** (e.g., 9B with two 7B parameter models). These models are often referred to as \"frankenmerges\" or \"Frankenstein models\" by the community.\nThis technique is very experimental, but it managed to create impressive models, like [goliath-120b](https://huggingface.co/alpindale/goliath-120b) using two Llama 2 70B models. The recently released [SOLAR-10.7B-v1.0](https://huggingface.co/upstage/SOLAR-10.7B-v1.0) also uses the same idea, called depth-up scaling [in their paper](https://arxiv.org/abs/2312.15166).\n_Example of configuration:_\nyaml\n\n```\nslices:\n - sources:\n - model: OpenPipe/mistral-ft-optimized-1218\n layer_range: [0, 32]\n - sources:\n - model: mlabonne/NeuralHermes-2.5-Mistral-7B\n layer_range: [24, 32]\nmerge_method: passthrough\ndtype: bfloat16\n```\n\nThe resulting frankenmerge will have all the 32 layers from the first model and 8 additional layers from the second model. This creates a frankenmerge with a total of 40 layers and 8.99B parameters. This config is inspired by [GML-Mistral-merged-v1](https://huggingface.co/zyh3826/GML-Mistral-merged-v1).\nYou can find the final model on the Hugging Face Hub at [mlabonne/NeuralPipe-9B-merged](https://huggingface.co/mlabonne/NeuralPipe-9B-merged).\n## 💻 Merge your own models\nIn this section, we will use mergekit to load a merge configuration, run it, and upload the resulting model to the Hugging Face Hub.\nFirst of all, we install mergekit directly from source as follows:\nbash\n\n```\n!git clone https://github.com/cg123/mergekit.git\n!cd mergekit && pip install -q -e .\n```\n\nIn the following block, we load the merge configuration in a YAML format. We also specify the name of the merged model for future use. You can copy/paste any configuration from the previous section here.\nThis time, we will use two different models: [Marcoroni-7B-v3](https://huggingface.co/AIDC-ai-business/Marcoroni-7B-v3) and [Mistral-7B-Merge-14-v0.1](https://huggingface.co/EmbeddedLLM/Mistral-7B-Merge-14-v0.1) and merge them with the SLERP method. We save the config as a yaml file to be used as input in the merge command.\npython\n\n```\nimport yaml\n\nMODEL_NAME = \"Marcoro14-7B-slerp\"\nyaml_config = \"\"\"\nslices:\n - sources:\n - model: AIDC-ai-business/Marcoroni-7B-v3\n layer_range: [0, 32]\n - model: EmbeddedLLM/Mistral-7B-Merge-14-v0.1\n layer_range: [0, 32]\nmerge_method: slerp\nbase_model: AIDC-ai-business/Marcoroni-7B-v3\nparameters:\n t:\n - filter: self_attn\n value: [0, 0.5, 0.3, 0.7, 1]\n - filter: mlp\n value: [1, 0.5, 0.7, 0.3, 0]\n - value: 0.5\ndtype: bfloat16\n\n\"\"\"\n\n# Save config as yaml file\nwith open('config.yaml', 'w', encoding=\"utf-8\") as f:\n f.write(yaml_config)\n```\n\nWe run the merge command with the following parameters:\n * `--copy-tokenizer` to copy the tokenizer from the base model\n * `--allow-crimes` and `--out-shard-size` to chunk the models into smaller shards that can be computed on a CPU with low RAM\n * `--lazy-unpickle` to enable the experimental lazy unpickler for lower memory usage\n\n\nIn addition, some models can require the `--trust_remote_code` flag (this is not the case with Mistral-7B).\nThis command will download the weights of all the models listed in the merge configuration and run the selected merge method (it should take ~10 minutes).\npython\n\n```\n# Merge models\n!mergekit-yaml config.yaml merge --copy-tokenizer --allow-crimes --out-shard-size 1B --lazy-unpickl\n```\n\nThe model is now merged and saved in the `merge` directory. Before uploading it, we can create a README file with all the information required for reproducibility. The following code block defines a Jinja template and automatically fills it with the data from the merge configuration.\npython\n\n```\n!pip install -qU huggingface_hub\n\nfrom huggingface_hub import ModelCard, ModelCardData\nfrom jinja2 import Template\n\nusername = \"mlabonne\"\n\ntemplate_text = \"\"\"\n---\nlicense: apache-2.0\ntags:\n- merge\n- mergekit\n- lazymergekit\n{%- for model in models %}\n- {{ model }}\n{%- endfor %}\n---\n\n# {{ model_name }}\n\n{{ model_name }} is a merge of the following models using [mergekit](https://github.com/cg123/mergekit):\n\n{%- for model in models %}\n* [{{ model }}](https://huggingface.co/{{ model }})\n{%- endfor %}\n\n## 🧩 Configuration\n\n```yaml\n{{- yaml_config -}}\n```\n\n\"\"\"\n# Create a Jinja template object\njinja_template = Template(template_text.strip())\n# Get list of models from config\ndata = yaml.safe_load(yaml_config) if \"models\" in data: models = [data[\"models\"][i][\"model\"] for i in range(len(data[\"models\"])) if \"parameters\" in data[\"models\"][i]] elif \"parameters\" in data: models = [data[\"slices\"][0][\"sources\"][i][\"model\"] for i in range(len(data[\"slices\"][0][\"sources\"]))] elif \"slices\" in data: models = [data[\"slices\"][i][\"sources\"][0][\"model\"] for i in range(len(data[\"slices\"]))] else: raise Exception(\"No models or slices found in yaml config\")\n# Fill the template\ncontent = jinja_template.render( model_name=MODEL_NAME, models=models, yaml_config=yaml_config, username=username, )\n# Save the model card\ncard = ModelCard(content) card.save('merge/README.md')\ntext\n\n```\n\nNow that we have a model card, we can push the entire folder to the Hub.\n\n```python\nfrom google.colab import userdata\nfrom huggingface_hub import HfApi\n\nusername = \"mlabonne\"\n\n# Defined in the secrets tab in Google Colab\napi = HfApi(token=userdata.get(\"HF_TOKEN\"))\n\napi.create_repo(\n repo_id=f\"{username}/{MODEL_NAME}\",\n repo_type=\"model\"\n)\napi.upload_folder(\n repo_id=f\"{username}/{MODEL_NAME}\",\n folder_path=\"merge\",\n)\n```\n\nThe model is now available on the Hugging Face Hub at [mlabonne/Marcoro14–7B-slerp](https://huggingface.co/mlabonne/Marcoro14-7B-slerp). In another notebook, we can try the model on a free T4 GPU using the following code:\npython\n\n```\n!pip install -qU transformers accelerate\n\nfrom transformers import AutoTokenizer\nimport transformers\nimport torch\n\nmodel = \"mlabonne/Marcoro14-7B-slerp\"\nmessages = [{\"role\": \"user\", \"content\": \"What is a large language model?\"}]\n\ntokenizer = AutoTokenizer.from_pretrained(model)\nprompt = tokenizer.apply_chat_template(\n messages,\n tokenize=False,\n add_generation_prompt=True\n)\npipeline = transformers.pipeline(\n \"text-generation\",\n model=model,\n torch_dtype=torch.float16,\n device_map=\"auto\",\n)\n\noutputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)\n```\n\nWe're asking the question \"What is a Large Language Model?\" and received this output:\n> _A large language model is a type of artificial intelligence (AI) system that has been trained on vast amounts of text data. It's designed to understand and generate human-like language, making predictions on what words or phrases might come next in a sentence or document. These models use complex algorithms and neural network architectures to learn from the data and improve their performance over time. Some well-known large language models include GPT-3 from OpenAI and BERT from Google._\nIt's looking good, but we need a more comprehensive evaluation. For this kind of general-purpose model, there are a few interesting benchmarks:\n * [**Chatbot Arena**](https://chat.lmsys.org/), which compiles an Elo-based LLM leaderboard based on human votes.\n * [**MT-bench**](https://chat.lmsys.org/) (same link), which uses GPT-4 as a judge to grade model responses on a set of multi-turn questions.\n * [**NousResearch benchmark suite**](https://github.com/teknium1/LLM-Benchmark-Logs), which aggregates four benchmarks: AGIEval, GPT4ALL, TruthfulQA, and Bigbench. GPT4ALL itself includes HellaSwag, OpenBookQA, Winogrande, ARC-Easy, ARC-Challenge, BoolQ, and PIQA.\n * [**Open LLM Leaderboard**](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), which aggregates six benchmarks: ARC, HellaSwag, MMLU, Winogrande, GSM8K, and TruthfulQA.\n\n\nUnfortunately, we can't submit our model to the Chatbot Arena. Instead, I chose to evaluate it using the Open LLM Leaderboard and NousResearch benchmarks.\nI submitted our model to the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) (\"🚀 Submit here!\" tab). As shown in the introduction, it ranked as **the best 7B parameter model** on the leaderboard. Here are the complete results:\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/0MZ-o1GRS9l4myMU5.png)\nImage by author\nThe problem with the Open LLM Leaderboard is that these benchmarks are public. It means that people can train LLMs on the test data to get better results. By merging the best models, we also contaminate our own results. It is safe to assume that **Marcoro14–7B-slerp is contaminated** and some models used in this merge have been trained on the test set. If you want to create the best model and not hack the leaderboard, I recommend only using non-merge models to create your own merges.\nThis is why we don't want to only rely on the OpenLLM Leaderboard. For NousResearch benchmark suite, I used [🧐 LLM AutoEval ](https://github.com/mlabonne/llm-autoeval)to compute the scores automatically with a simple Colab notebook. Here are the results compared to the excellent O[penHermes-2.5-Mistral-7B:](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B)\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/0UOADh7Em1iNOssOv.png)\nImage by author\nWe get a significant improvement over this model on **every benchmark**. Note that NousResearch benchmark suite shares some tasks with the Open LLM Leaderboard: ARC-Challenge, TruthfulQA, HellaSwag, and Winogrande. To the best of my knowledge, Bigbench is the only benchmark that is 100% different (feel free to contact me if that's not the case). However, one of the models we used in this merge could still have been trained on Bigbench.\n## Conclusion\nIn this article, we introduced the concept of merging LLMs with four different methods. We detailed how SLERP, TIES, DARE, and passthrough work and provided examples of configurations. Finally, we ran SLERP with mergekit to create [Marcoro14–7B-slerp](https://huggingface.co/mlabonne/Marcoro14-7B-slerp) and upload it to the Hugging Face Hub. We obtained excellent performance on two benchmark suites: Open LLM Leaderboard (**best-performing 7B model**) and NousResearch. If you want to create your own merges, I recommend using my automated notebook [🥱 LazyMergekit.](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing)\nAnother way of combining multiple models is to merge them in a Mixture of Experts (MoE) architecture. In the next article, we'll discuss how to do this in detail and create our [own Mixtral-like model](https://huggingface.co/mlabonne/Beyonder-4x7B-v2). If you liked this article, please follow me on Medium and Twitter [@maximelabonne](https://twitter.com/maximelabonne).\n* * *\n_Learn more about machine learning and support my work with one click - become a Medium member here:_\n> [**Join Medium with my referral link - Maxime Labonne**](https://medium.com/@mlabonne/membership)\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657499317_sxy3ci.png)\nWritten By\n## [Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[See all from Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Data Science](https://towardsdatascience.com/tag/data-science/)[Large Language Models](https://towardsdatascience.com/tag/large-language-models/)[Programming](https://towardsdatascience.com/tag/programming/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fmerge-large-language-models-with-mergekit-2118fb392b54%2F&title=Merge%20Large%20Language%20Models%20with%20mergekit)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fmerge-large-language-models-with-mergekit-2118fb392b54%2F&title=Merge%20Large%20Language%20Models%20with%20mergekit)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fmerge-large-language-models-with-mergekit-2118fb392b54%2F&text=Merge%20Large%20Language%20Models%20with%20mergekit)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/11/1oi4CJU_L-Zl-YSolShHQlw.png)](https://towardsdatascience.com/understanding-predictive-maintenance-wave-data-feature-engineering-part-1-fd22b6a19323/)\n## [Understanding Predictive Maintenance - Wave Data: Feature Engineering (Part 1)](https://towardsdatascience.com/understanding-predictive-maintenance-wave-data-feature-engineering-part-1-fd22b6a19323/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nAll the information you need to start learning about wave-data signal processing\n[Marcin Stasko](https://towardsdatascience.com/author/marcin-staskopl/)\nNovember 21, 202316 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/02/0Jku-dbfbNjM5_2m1-scaled.jpg)](https://towardsdatascience.com/introduction-to-r-318ca6dcb56/)\n## [Introduction to R](https://towardsdatascience.com/introduction-to-r-318ca6dcb56/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nIs R the right language for me?\n[Mike Wolfe](https://towardsdatascience.com/author/miketechgame/)\nFebruary 18, 20216 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/13pPeo8VPDVQeUG6fN4fb1w-scaled.jpeg)](https://towardsdatascience.com/17-quick-examples-to-learn-python-lists-196070a055c3/)\n## [17 Quick Examples to Learn Python Lists](https://towardsdatascience.com/17-quick-examples-to-learn-python-lists-196070a055c3/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nOne of the most commonly used Python data structures Data structures are the key components...\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nMay 21, 20216 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/0zYJaVap8uvMIxSL4-scaled.jpg)](https://towardsdatascience.com/python-tuple-named-tuples-edaee4d1b191/)\n## [Python \"Tuple+\": Named Tuples](https://towardsdatascience.com/python-tuple-named-tuples-edaee4d1b191/)\n[Programming](https://towardsdatascience.com/category/programming/)\nTuples are a powerful Python type - but named tuples even more so!\n[Marcin Kozak](https://towardsdatascience.com/author/nyggus/)\nJanuary 3, 202427 min read\n * [![](https://towardsdatascience.com/images/fallback-image.jpg)](https://towardsdatascience.com/better-pythoning-2-list-comprehensions-including-nlp-examples-with-spacy-88cd76fc6412/)\n## [Better Pythoning 2: List Comprehensions (including NLP examples with spaCy)](https://towardsdatascience.com/better-pythoning-2-list-comprehensions-including-nlp-examples-with-spacy-88cd76fc6412/)\n[NLP](https://towardsdatascience.com/category/artificial-intelligence/nlp/)\nHow to shorten lines of code and make it more readable in a few simple steps\n[Yousef Nami](https://towardsdatascience.com/author/namiyousef96/)\nFebruary 19, 20216 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/1ZcCHZmCz4aDx4i99l32z-A-scaled.jpeg)](https://towardsdatascience.com/data-science-workflows-notebook-to-production-26afc13442bb/)\n## [Data Science Workflows - Notebook to Production](https://towardsdatascience.com/data-science-workflows-notebook-to-production-26afc13442bb/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nReduce the time from research to production using various MLOps tools and techniques.\n[Nir Barazida](https://towardsdatascience.com/author/nirbarazida/)\nNovember 24, 202111 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/07/1fxVRrvmFa5u0g1gqsoOWgg.png)](https://towardsdatascience.com/word-embedding-in-nlp-one-hot-encoding-and-skip-gram-neural-network-81b424da58f2/)\n## [Word Embedding in NLP: One-Hot Encoding and Skip-Gram Neural Network](https://towardsdatascience.com/word-embedding-in-nlp-one-hot-encoding-and-skip-gram-neural-network-81b424da58f2/)\n[NLP](https://towardsdatascience.com/category/artificial-intelligence/nlp/)\nI'm a poet-turned-programmer who has just begun learning about the wonderful world of natural language processing. In this post, I'll be...\n[Jianna Park](https://towardsdatascience.com/author/jiannapark/)\nJuly 24, 202010 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/0EnRawczB5k_0TE2m.jpg)](https://towardsdatascience.com/speed-up-pandas-code-with-numpy-fdee76210794/)\n## [Speed up Pandas code with Numpy](https://towardsdatascience.com/speed-up-pandas-code-with-numpy-fdee76210794/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nBut I can't vectorise this, can I? .... yes, you probably can!\n[Thomas Reid](https://towardsdatascience.com/author/thomas_reid/)\nJanuary 13, 202514 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/1wPFJ6xWejfIDLmIDlJPIwA.jpeg)](https://towardsdatascience.com/exploring-the-latent-space-of-your-convnet-classifier-b6eb862e9e55/)\n## [Exploring the Latent Space of a ConvNet Image Classifier](https://towardsdatascience.com/exploring-the-latent-space-of-your-convnet-classifier-b6eb862e9e55/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nLinearity in the latent space of hidden layers!\n[Sachin](https://towardsdatascience.com/author/smodi93/)\nOctober 2, 20206 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500) Check Your Biases ](https://towardsdatascience.com/check-your-biases-ba2685a7f799/ \"Check Your Biases\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500) Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads ](https://towardsdatascience.com/machine-learning-tricks-learning-with-chatgpt-and-other-recent-must-reads-f277648a1921/ \"Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads\")\n\n\nword word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word\n \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1\n", "metadata": { "title": "Merge Large Language Models with mergekit | Towards Data Science", "description": "Create your own models easily, no GPU required!", "keywords": null, "author": "Maxime Labonne", "og:locale": "en_US", "og:type": "article", "og:title": "Merge Large Language Models with mergekit", "og:description": "Create your own models easily, no GPU required!", "og:url": "https://towardsdatascience.com/merge-large-language-models-with-mergekit-2118fb392b54/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/1kBMm6Dgfej-snEAC7dXDjg.jpeg", "og:image:width": "1792", "og:image:height": "1008", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Maxime Labonne", "twitter:label2": "Est. reading time", "twitter:data2": "11 min read", "article:published_time": "2024-01-08T03:17:00.000Z", "article:modified_time": "2024-01-08T03:17:00.000Z", "article:author": "https://towardsdatascience.com/author/mlabonne/" }, "paywall": { "listed_paid": true, "detected": false, "marker": "", "public_only": true, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "", "retried": true }, "code_blocks": [ { "language": "txt", "file": "attachments/3409cf6aecf7/code/block_000.txt", "chars": 410 }, { "language": "txt", "file": "attachments/3409cf6aecf7/code/block_001.txt", "chars": 392 }, { "language": "txt", "file": "attachments/3409cf6aecf7/code/block_002.txt", "chars": 480 }, { "language": "txt", "file": "attachments/3409cf6aecf7/code/block_003.txt", "chars": 229 }, { "language": "txt", "file": "attachments/3409cf6aecf7/code/block_004.txt", "chars": 93 }, { "language": "txt", "file": "attachments/3409cf6aecf7/code/block_005.txt", "chars": 583 }, { "language": "txt", "file": "attachments/3409cf6aecf7/code/block_006.txt", "chars": 115 }, { "language": "txt", "file": "attachments/3409cf6aecf7/code/block_007.txt", "chars": 523 }, { "language": "txt", "file": "attachments/3409cf6aecf7/code/block_008.txt", "chars": 813 }, { "language": "python", "file": "attachments/3409cf6aecf7/code/block_009.py", "chars": 349 }, { "language": "txt", "file": "attachments/3409cf6aecf7/code/block_010.txt", "chars": 619 } ], "images": [ { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/1kBMm6Dgfej-snEAC7dXDjg.jpeg", "alt": "Image by author", "local_path": "attachments/3409cf6aecf7/images/000.jpeg", "bytes": 275666, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/1Gm1kcCkX76hMD7dMK1AJ7Q.png", "alt": "Image by author", "local_path": "attachments/3409cf6aecf7/images/001.png", "bytes": 48216, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/0MZ-o1GRS9l4myMU5.png", "alt": "Image by author", "local_path": "attachments/3409cf6aecf7/images/002.png", "bytes": 6568, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/0UOADh7Em1iNOssOv.png", "alt": "Image by author", "local_path": "attachments/3409cf6aecf7/images/003.png", "bytes": 13467, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/11/1oi4CJU_L-Zl-YSolShHQlw.png", "alt": "", "local_path": "attachments/3409cf6aecf7/images/004.png", "bytes": 1576898, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/02/0Jku-dbfbNjM5_2m1-scaled.jpg", "alt": "", "local_path": "attachments/3409cf6aecf7/images/005.jpg", "bytes": 473287, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/13pPeo8VPDVQeUG6fN4fb1w-scaled.jpeg", "alt": "", "local_path": "attachments/3409cf6aecf7/images/006.jpeg", "bytes": 883327, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/0zYJaVap8uvMIxSL4-scaled.jpg", "alt": "", "local_path": "attachments/3409cf6aecf7/images/007.jpg", "bytes": 393866, "downloaded": true }, { "src": "/images/fallback-image.jpg", "alt": "", "local_path": "", "bytes": 0, "downloaded": false }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/1ZcCHZmCz4aDx4i99l32z-A-scaled.jpeg", "alt": "", "local_path": "attachments/3409cf6aecf7/images/009.jpeg", "bytes": 809266, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/07/1fxVRrvmFa5u0g1gqsoOWgg.png", "alt": "", "local_path": "attachments/3409cf6aecf7/images/010.png", "bytes": 139100, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/0EnRawczB5k_0TE2m.jpg", "alt": "", "local_path": "attachments/3409cf6aecf7/images/011.jpg", "bytes": 341332, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/1wPFJ6xWejfIDLmIDlJPIwA.jpeg", "alt": "", "local_path": "attachments/3409cf6aecf7/images/012.jpeg", "bytes": 219071, "downloaded": true } ], "audio": [], "videos": [], "links": [], "retry_at": "2026-10-03T07:59:12", "repair_version": 2, "repaired_at": "2026-10-14T11:11:48Z", "source_url": "https://towardsdatascience.com/merge-large-language-models-with-mergekit-2118fb392b54/", "delivery_status": "excluded_paid", "discovery_status": "excluded_paid", "missing_reason": "按原约定排除付费/会员全文" }