{ "source_file": "0930", "source_platform": "研究机构", "author": "Hugging Face Blog", "title_listed": "Mixture of Experts (MoEs) in Transformers", "keywords": "MoE,Transformer,架构", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://huggingface.co/blog/moe-transformers", "publish_date": "2024-02", "corpus_stage": "预训练/SFT", "notes": "Transformer中的混合专家架构", "id": "e44d72287f61", "collected_at": "2026-10-02T15:23:32", "title": "Mixture of Experts (MoEs) in Transformers", "body_markdown": "Back to Articles\nMixture of Experts (MoEs) in Transformers\nPublished\n\t\t\t\t\tFebruary 26, 2026\nUpdate on GitHub\nUpvote\n180\n+174\nAritra Roy Gosthipaty\nariG23498\nFollow\nPedro Cuenca\npcuenq\nFollow\nmerve\nmerve\nFollow\nIlyas Moutawwakil\nIlyasMoutawwakil\nFollow\nArthur Zucker\nArthurZ\nFollow\nSergio Paniego\nsergiopaniego\nFollow\nPablo Montalvo\nMolbap\nFollow\nIntroduction\nOver the past few years, scaling dense language models has driven most progress in LLMs. From early models like the original\nULMFiT\n(~30M parameters) or GPT-2 (1.5B parameters, which at the time was considered \"too dangerous to release\" 🧌), and eventually to today’s hundred-billion–parameter systems, the recipe was simple:\nMore data + more parameters gives better performance.\nScaling laws\nreinforced this trend, but dense scaling has practical limits:\nTraining becomes increasingly expensive.\nInference latency grows.\nDeployment requires significant memory and hardware.\nThis is where Mixture of Experts (MoEs) enter the picture.\nIf you're already familiar with MoEs and want to jump straight into the engineering work done in transformers, you can head directly to\nTransformers and MoEs\n.\nFrom Dense to Sparse: What Are MoEs?\nA Mixture of Experts model keeps the Transformer backbone, but replaces certain dense feed-forward layers with a set of\nexperts\n. An “expert” is not a topic-specialized module (e.g., \"math expert\", \"code expert\"). It is simply a learnable sub-network. For each token, a\nrouter\nselects a small subset of experts to process it.\nFigure 1: Expert 1 among 4 experts is activated (Source:\nMaarten Grootendorst\n)\nDifferent tokens activate different experts, based on their hidden representations.\nModel capacity depends on total parameters, but inference speed depends on active parameters.\nThis is the key idea.\nFor example, take\ngpt-oss-20b\n. It has 21B total parameters, but uses 4 active experts per token, out of a total of 32 experts. Considering the shared components plus the active experts, this model uses ~3.6B active parameters per token. Running this model on an M3 Ultra Mac, which has a memory bandwidth of about 800 GB, we could estimate generation speed as ~\n800 / (3.6 * 2)\nin\nbfloat16\n, where each parameter takes 2 bytes. This yields about\n111 tokens per second\n. The actual performance number we get is ~115 tok/s, which is very close to the back-of-the-envelope calculation.\nYour browser does not support the video tag.\nThis super fast speed confirms the model works approximately as a 3.6B parameter one, but it has the same capacity (or quality) as a 21B parameter model.\n(Note: speed would be even faster if we used kernels for the native mxfp4 quantization the model uses).\nMoEs are attractive for these reasons:\nBetter Compute Efficiency\nGiven a fixed training FLOP budget, MoEs often outperform dense counterparts.\nFigure 2: Dense vs. MoE training curves (Source:\nOLMoE: Open Mixture-of-Experts Language Models\n)\nThis means faster iteration and better scaling efficiency.\nA Natural Parallelization Axis\nExperts provide a structural boundary in the computation graph. Since different tokens engage different experts, we can parallelize across experts (we discuss this later in\nExpert Parallelism\n).\nIndustry Adoption\nRecent major MoE releases of open models that happened in the past few weeks include\nQwen 3.5\n,\nMiniMax M2\n,\nGLM-5\n, or\nKimi K2.5\n.\nThe trend accelerated after the success of\nDeepSeek R1\nin January 2025, building on earlier systems like\nDeepSeek V2\n. Another early MoE was\nMixtral-8x7B\n, released in December 2023.\nFigure 3: 2-year timeline of MoE model addition to the\ntransformers\nlibrary. DeepSeek R1 marks a clear inflection point.\nClosed labs use MoEs too. ChatGPT has long been\nrumored\nto use a sparse architecture, and the open\ngpt-oss models\ncertainly do.\nIf you want to learn more about MoEs in general, we strongly suggest reading\nthis blog\nand watching our recent\nYouTube video on routing\n.\nTransformers and MoEs\nMost tooling in the ecosystem, including model loading, device placement, quantization, and backend execution was originally designed for\ndense\nmodels. MoEs challenge these assumptions.\nMaking MoEs\nfirst-class citizens\nin\ntransformers\nmeans redesigning parts of the loading pipeline, execution model, and distributed abstractions, not just adding new model classes. We’ll focus on how the\ntransformers\nlibrary has evolved to support sparse architectures across:\nWeight Loading Refactor\nExpert Backend\nExpert Parallelism\nTraining MoEs with transformers\nWeight Loading Refactor\nAutoModelForCausalLM.from_pretrained(\"model_id\")\ndownloads and loads model weights into a PyTorch model. For dense models, loading is relatively straightforward where each tensor in the checkpoint maps one-to-one to a parameter in the runtime module.\nFor MoEs, it’s more complicated. In most MoE checkpoints, each expert is serialized independently. If you peek inside the\nDeepSeek-V3 checkpoint index\n, you’ll see keys like:\nmodel.layers.3.mlp.experts.0.gate_proj.weight\n...\nmodel.layers.3.mlp.experts.255.gate_proj.weight\nEach expert has its own set of weight matrices, essentially 256 (0 to 255 total, taking DeepSeek-V3 as an example) small feed-forward networks saved side by side. At runtime, however, GPUs execute optimized kernels. Modern MoE kernels such as\ngrouped GEMMs and fused MoE implementations\nare designed to process\nall experts in a single operation\n, not by looping over them one at a time.\nTo do that efficiently, they require expert weights to be packed into a single\ncontiguous tensor\n.\nSo we have a mismatch:\nCheckpoint:\n256 separate tensors\nRuntime:\n1 packed tensor\nBridging this gap systematically is what the\nweight loading refactor\nenables.\nWith the introduction of a\ngeneric WeightConverter\n, the mental model shifted from:\nA checkpoint already matches my runtime layout; loading is mostly a key-by-key copy.\nto:\nA checkpoint is just a serialized source of tensors. Loading is a\nconversion pipeline\nthat transforms them into the runtime layout we want.\nDynamic Weight Loading with\nWeightConverter\nThe central abstraction introduced by this refactor is\ndynamic weight loading\nvia a\nWeightConverter\n.\nWeightConverter\nlets us define:\nsource key patterns → target key(s) + operations\nPrimitive operations (chunk, concatenate, etc.) are composable. Two that are particularly useful for MoEs:\nMergeModulelist\nmerges a list of tensors into a single tensor. For example, you can compose\nMergeModulelist\nwith\nConcatenate\nto stack the experts in a MoE and pack them into one tensor.\nWeightConverter(\n [\n\"block_sparse_moe.experts.*.w1.weight\"\n,\n\"block_sparse_moe.experts.*.w3.weight\"\n,],\n\"mlp.experts.gate_up_proj\"\n,\n operations=[\n MergeModulelist(dim=\n0\n),\n Concatenate(dim=\n1\n),\n ],\n)\nSplitModulelist\nsplits a tensor back into a list of tensors. For example, you can split a stack of experts back into individual experts.\nWeightConverter(\n\"mlp.experts.down_proj\"\n,\n\"block_sparse_moe.experts.*.w2.weight\"\n,\n operations=[SplitModulelist(dim=\n0\n)],\n)\nLazy Materialization of Tensors\nThe refactor improves not just\nwhat\nconversions exist, but\nhow\nthey’re scheduled.\nThe loader scans checkpoint keys once, matches them against converter patterns, and groups tensors per converter. Once a key is identified as needed, it’s registered as a\nfuture\nand materialized via a thread pool. Conversion operations run only once their dependencies are ready. For example,\nMergeModulelist\nwaits until all experts for a layer are loaded.\nThis avoids repeated scans and reduces memory peaks.\nBenchmark: Weight-Loading Pipeline Improvements\nTo evaluate the improvements introduced by the new weight-loading pipeline, we benchmarked the v4 vs v5 versions of\ntransformers\n. The focus is on loading speed of large MoE models, which is often a bottleneck in training and inference.\nWe benchmarked v4 vs v5 using:\nv4 branch:\nhttps://github.com/ariG23498/transformers/tree/bench-v4\nv5 branch:\nhttps://github.com/ariG23498/transformers/tree/bench-v5\nExample:\nfrom\ntransformers\nimport\nAutoModelForCausalLM\n\nmodel_id =\n\"Qwen/Qwen1.5-110B-Chat\"\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\nTwo relevant environment variables:\nHF_ENABLE_PARALLEL_LOADING\n: Enables parallel shard loading via threads.\nHF_DEACTIVATE_ASYNC_LOAD\n:Disables the new async pipeline (v5 escape hatch).\nResults\nModel:\nQwen/Qwen1.5-110B-Chat\nGPU:\n1× A100 (80GB)\nVersion\nStrategy\nLoading Mode\nTime\nv4.57.6\ndevice_map=\"auto\"\nThreadpool\n66.24s\nv4.57.6\ndevice_map=\"auto\"\nSequential\n67.29s\nv4.57.6\nTP\n—\nOOM\nv5\ndevice_map=\"auto\"\nAsync (default)\n20.71s\nv5\ndevice_map=\"auto\"\nSync\n45.3s\nv5\nTP\nAsync\n10.1s\nv5\nTP\nSync\n19.28s\nFigure 4: Loading benchmarks (v4 vs v5)\nThe speedup is not just “more threads.”\nIt’s the combination of\nSingle-pass routing\n,\nAsync materialization\n, and\nConversion-aware scheduling\nwhich together avoid unnecessary materialization and memory peaks while enabling expert packing and projection fusion at load time.\nWhere Quantization Fits In\nWith this refactor we can now create the runtime module structure first and then convert the weights into the structure. We can now optionally attach quantization within the conversion pipeline, making quantization part of the weight loading pipeline itself. This is crucial because quantizing “per expert” only makes sense once experts exist in a predictable packed layout.\nThis end to end pipeline was not possible earlier and now it comes to the users as an exposed API.\nExpert Backend\nOnce experts are packed into a single runtime tensor, another question arises:\nHow do you actually route through them efficiently?\nIn a Mixture of Experts model, each token is routed to different experts. This means the runtime must dispatch tokens to their selected expert weights, execute the projections efficiently, apply the routing weights and then collect and reorder the results.\nThis is what the\nExperts Backend system\n(introduced in\nPR #42697\n) addresses. The Experts Backend introduces a\npluggable execution architecture\nthat decouples expert computation from the model implementation. Instead of hardcoding one dispatch strategy inside each MoE model, the system allows expert layers to dynamically select a backend at runtime.\nThis is implemented via a decorator pattern:\n@use_experts_implementation\nThe decorator wraps expert classes and dispatches computation to the selected backend automatically.\nThree backends are currently provided:\neager\nwhich loops over the selected experts and applies projections per expert. This is used for correctness reference and debugging.\nbatched_mm\nuses the\ntorch.bmm\nAPI. This duplicate selected expert weights per token and performs a single batched GEMM. This backend is very well suited for small batch, GPU-heavy workloads where memory is available.\ngrouped_mm\nuses\ntorch._grouped_mm\nAPI. Here we sort tokens by expert ID, group them, and then perform a single grouped GEMM. This backend shines with large batches or memory-constrained setups.\nFigure: Expert backend illustration\nExpert Parallelism\nMixture of Experts (MoE) models can have hundreds of billions of parameters (far more than what fits on a single GPU). Expert parallelism (EP) addresses this by distributing experts across multiple devices. Each device loads only its assigned subset of experts, computes for those experts and then participates in result aggregation. This approach scales models to far larger parameter counts without increasing computation cost because each token activates only a few experts.\nExpert parallelism is enabled via\nenable_expert_parallel\n:\nimport\ntorch\nfrom\ntransformers\nimport\nAutoModelForCausalLM, AutoTokenizer\nfrom\ntransformers.distributed.configuration_utils\nimport\nDistributedConfig\n\ndistributed_config = DistributedConfig(enable_expert_parallel=\nTrue\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n\"openai/gpt-oss-120b\"\n,\n dtype=\n\"auto\"\n,\n distributed_config=distributed_config,\n)\nLaunch with:\ntorchrun --nproc-per-node N script.py\nWhere\nN\nevenly divides the total number of experts, and possibly matches the number of GPUs in your node.\nWhen\nenable_expert_parallel=True\n, the model switches from the standard tensor-parallel (TP) plan to an expert-parallel (EP) plan with specialized sharding strategies.\nCore components of EP lie in:\nGroupedGemmParallel\n: This splits the expert weights along the expert dimension (\ndim=0\n). Here each device loads only\nnum_experts / num_devices\n.\nRouterParallel\n: This remaps global expert indices to local indices, masks out experts not assigned to the current rank, ensures each device computes only with its local experts and uses an all-reduce to combine partial outputs across devices.\nTraining MoEs with Transformers\nMoEs are excellent for scaling inference, but training them is significantly more complex.\nMoEs have a Massive parameter count, the distributed expert communication is complicated, there are routing in-stabilities that need to be handled. To address this, we collaborated with\nUnsloth\nto enable significantly faster Mixture-of-Experts training:\n~12× faster MoE training\n>35% VRAM reduction\n~6× longer context\n12–30× overall speedup compared to v4\nWe leverage the Expert Backend abstraction, standardize around PyTorch’s\ntorch._grouped_mm\nAPI and use custom Triton grouped-GEMM + LoRA kernels. Unsloth builds on top of the Transformers (and TRL) optimizations to push performance further.\nFor full details, we recommend reading:\nUnsloth’s official guide\nConclusion\nAs sparse architectures continue to evolve, we want the transformers library to evolve with them. If you’re building with MoEs or experimenting with new sparse ideas, we’d love to hear from you. Let us know what abstractions, kernels, or workflows you’d like to see next in\ntransformers\n.\nModels mentioned in this article\n6\nPapers mentioned in this article\n2\nCollections mentioned in this article\n5\nMore Articles from our Blog\nvllm\ntransformers\ninference\nNative-speed vLLM transformers modeling backend\n77\nJuly 8, 2026\ntransformers\npytorch\noptimization\nUnlocking asynchronicity in continuous batching\n68\nMay 14, 2026\nCommunity\nbambuuai\nMar 8\nThe\nenable_expert_parallel\nflag hiding the complexity of\nGroupedGemmParallel\n+\nRouterParallel\nbehind a single config is a great DX win — distributing experts across devices used to require a lot of custom plumbing.\nSee translation\n🔥\n1\n1\n+\nReply\nEdit\nPreview\nUpload images, audio, and videos by dragging in the text input, pasting, or\nclicking here\n.\nTap or paste here to upload images\nComment\n·\nSign up\nor\nlog in\nto comment\nUpvote\n180\n+168\nModels mentioned in this article\n6\nPapers mentioned in this article\n2\nCollections mentioned in this article\n5", "metadata": { "title": "Mixture of Experts (MoEs) in Transformers", "description": "We’re on a journey to advance and democratize artificial intelligence through open source and open science.", "keywords": null, "author": null, "og:title": "Mixture of Experts (MoEs) in Transformers", "og:description": "We’re on a journey to advance and democratize artificial intelligence through open source and open science.", "og:type": "website", "og:url": "https://huggingface.co/blog/moe-transformers", "og:image": "https://huggingface.co/blog/assets/moe-transformers/thumbnail.png", "twitter:card": "summary_large_image", "twitter:site": "@huggingface", "twitter:image": "https://huggingface.co/blog/assets/moe-transformers/thumbnail.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/e44d72287f61/code/body-000.txt", "chars": 98, "language": "text" }, { "file": "attachments/e44d72287f61/code/body-001.txt", "chars": 49, "language": "text" }, { "file": "attachments/e44d72287f61/code/body-002.txt", "chars": 222, "language": "text" }, { "file": "attachments/e44d72287f61/code/body-003.txt", "chars": 133, "language": "text" }, { "file": "attachments/e44d72287f61/code/body-004.txt", "chars": 138, "language": "text" }, { "file": "attachments/e44d72287f61/code/body-005.txt", "chars": 28, "language": "text" }, { "file": "attachments/e44d72287f61/code/body-006.txt", "chars": 355, "language": "text" }, { "file": "attachments/e44d72287f61/code/body-007.txt", "chars": 38, "language": "text" } ], "images": [ { "status": "ok", "url": "https://huggingface.co/avatars/70862604ca512a504b00ade732c94cd5.svg", "bytes": 397, "reused": false, "local_path": "attachments/e44d72287f61/images/body-106caaf8486c4d86173b.svg", "src": "https://huggingface.co/avatars/70862604ca512a504b00ade732c94cd5.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Ao0HtSsM1-F3dab_mWU0D.png", "bytes": 9032, "reused": false, "local_path": "attachments/e44d72287f61/images/body-4157c3911de529d25d7d.png", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Ao0HtSsM1-F3dab_mWU0D.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/I232_-tiiUTXUnlIRxeJD.png", "bytes": 11778, "reused": false, "local_path": "attachments/e44d72287f61/images/body-f0013fbe4974416317a5.png", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/I232_-tiiUTXUnlIRxeJD.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/1c633aebe6352d84a27808d2c75ad4d9.svg", "bytes": 1282, "reused": false, "local_path": "attachments/e44d72287f61/images/body-f0c8ea82b26d4379c6ab.svg", "src": "https://huggingface.co/avatars/1c633aebe6352d84a27808d2c75ad4d9.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/b7b799648342835b330e9914865b4428.svg", "bytes": 397, "reused": false, "local_path": "attachments/e44d72287f61/images/body-6db91804c25486504147.svg", "src": "https://huggingface.co/avatars/b7b799648342835b330e9914865b4428.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/056217a2b6fe0dafd3ac1a553f8b4098.svg", "bytes": 397, "reused": false, "local_path": "attachments/e44d72287f61/images/body-28a0fb2c9bba59a3b8a3.svg", "src": "https://huggingface.co/avatars/056217a2b6fe0dafd3ac1a553f8b4098.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/moe_routing.png", "bytes": 30423, "reused": true, "local_path": "attachments/e44d72287f61/images/007.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/moe_routing.png", "alt": "MoE routing diagram", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/faster_training.png", "bytes": 253152, "reused": true, "local_path": "attachments/e44d72287f61/images/008.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/faster_training.png", "alt": "MoE vs Dense training graphs", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/moe_2y_timeline.png", "bytes": 105650, "reused": true, "local_path": "attachments/e44d72287f61/images/009.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/moe_2y_timeline.png", "alt": "2-year timeline of MoE model addition in the transformers package", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/loading_benchmark.png", "bytes": 162052, "reused": true, "local_path": "attachments/e44d72287f61/images/010.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/loading_benchmark.png", "alt": "Loading benchmarks", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/expert_backend.png", "bytes": 227038, "reused": true, "local_path": "attachments/e44d72287f61/images/011.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/expert_backend.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/blog/assets/native-speed-vllm-transformers-backend/thumbnail.png", "bytes": 6259641, "reused": false, "local_path": "attachments/e44d72287f61/images/body-9de8868c9fd090f0315d.png", "src": "https://huggingface.co/blog/assets/native-speed-vllm-transformers-backend/thumbnail.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ov-55uEKP4mJhavhdQ42x.png", "bytes": 6084, "reused": false, "local_path": "attachments/e44d72287f61/images/body-18d6cc7e6086dfe62195.png", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ov-55uEKP4mJhavhdQ42x.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/5e3aec01f55e2b62848a5217/PMKS0NNB4MJQlTSFzh918.jpeg", "bytes": 11598, "reused": false, "local_path": "attachments/e44d72287f61/images/body-741cc42f07a702f49510.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/5e3aec01f55e2b62848a5217/PMKS0NNB4MJQlTSFzh918.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/blog/assets/continuous_async/thumbnail.png", "bytes": 2433012, "reused": false, "local_path": "attachments/e44d72287f61/images/body-60c6d213869210179d0b.png", "src": "https://huggingface.co/blog/assets/continuous_async/thumbnail.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/6123945a0ed258ebc83f3d56/8wMHFQHEV24G_ljl4kPxQ.jpeg", "bytes": 9028, "reused": false, "local_path": "attachments/e44d72287f61/images/body-7877b5dec08e4fff29fb.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/6123945a0ed258ebc83f3d56/8wMHFQHEV24G_ljl4kPxQ.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1617264212503-603d25b75f9d390ab190b777.jpeg", "bytes": 9492, "reused": true, "local_path": "attachments/e44d72287f61/images/001.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1617264212503-603d25b75f9d390ab190b777.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/608aabf24955d2bfc3cd99c6/-YxmtpzEmf3NKOTktODRP.jpeg", "bytes": 9712, "reused": true, "local_path": "attachments/e44d72287f61/images/000.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/608aabf24955d2bfc3cd99c6/-YxmtpzEmf3NKOTktODRP.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/69a628eeaa5258e50d8f3854/EiWSeNiHNTpBTDmjC9kD2.png", "bytes": 9774, "reused": false, "local_path": "attachments/e44d72287f61/images/body-6abd3d2dcf1066b3f824.png", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/69a628eeaa5258e50d8f3854/EiWSeNiHNTpBTDmjC9kD2.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/6a10464a77bcc77c7530e91b/quAV_GY8xbLx89fEVtJ85.jpeg", "bytes": 11720, "reused": false, "local_path": "attachments/e44d72287f61/images/body-dc474aaa11695fa58467.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/6a10464a77bcc77c7530e91b/quAV_GY8xbLx89fEVtJ85.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/414aebdde48fa58952b452d550c47264.svg", "bytes": 1280, "reused": false, "local_path": "attachments/e44d72287f61/images/body-2f5ecfe1d23c4ac55a2b.svg", "src": "https://huggingface.co/avatars/414aebdde48fa58952b452d550c47264.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1594651707950-noauth.jpeg", "bytes": 6664, "reused": false, "local_path": "attachments/e44d72287f61/images/body-9d765f86233ead48ab54.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1594651707950-noauth.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/64e8ea3892d9db9a93580fe3/yFASo13WEQPUn3xzF280I.jpeg", "bytes": 6424, "reused": false, "local_path": "attachments/e44d72287f61/images/body-9514c7ab61be045d390b.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/64e8ea3892d9db9a93580fe3/yFASo13WEQPUn3xzF280I.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/62d648291fa3e4e7ae3fa6e8/oatOwf8Xqe5eDbCSuYqCd.png", "bytes": 4308, "reused": false, "local_path": "attachments/e44d72287f61/images/body-2f7cd9aaae2da43f9642.png", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/62d648291fa3e4e7ae3fa6e8/oatOwf8Xqe5eDbCSuYqCd.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/620b7c408f5871b8a1a168a7/49M2lucv3I24rOMJFnhVd.jpeg", "bytes": 13562, "reused": false, "local_path": "attachments/e44d72287f61/images/body-ff6dc7412388fb865606.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/620b7c408f5871b8a1a168a7/49M2lucv3I24rOMJFnhVd.jpeg", "alt": "", "downloaded": true } ], "audio": [], "videos": [ { "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/gpt-oss-20-inference.mp4", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/gpt-oss-20-inference.mp4", "status": "invalid_or_wrong_resolution", "kind": "direct_video", "bytes": 1015808, "reused": true, "content_type": "video/mp4", "valid": true, "height": 1000, "width": 1600, "duration": "8.733000", "local_path": "attachments/e44d72287f61/videos/repair-7444ea4bc10e830cdcda.mp4", "title": "", "retried_at": "2026-10-20T17:42:14Z" }, { "url": "https://www.youtube.com/watch?v=CDnkFbW-uEQ", "src": "https://youtu.be/CDnkFbW-uEQ", "status": "ok", "kind": "single_video", "local_path": "attachments/e44d72287f61/videos/repair-CDnkFbW-uEQ.mp4", "bytes": 132917845, "valid": true, "height": 1080, "width": 1920, "duration": "2054.421000", "reused": false } ], "links": [ { "href": "https://huggingface.co/blog", "label": "Back to Articles", "type": "a" }, { "href": "https://github.com/huggingface/blog/blob/main/moe-transformers.md", "label": "Update on GitHub", "type": "a" }, { "href": "https://huggingface.co/login?next=%2Fblog%2Fmoe-transformers", "label": "Upvote 180", "type": "a" }, { "href": "https://huggingface.co/leocnj", "label": "", "type": "a" }, { "href": "https://huggingface.co/samudraneel05", "label": "", "type": "a" }, { "href": "https://huggingface.co/gawadx1", "label": "", "type": "a" }, { "href": "https://huggingface.co/HoangLayor", "label": "", "type": "a" }, { "href": "https://huggingface.co/zz-user", "label": "", "type": "a" }, { "href": "https://huggingface.co/codenamicsDBX", "label": "", "type": "a" }, { "href": "https://huggingface.co/ariG23498", "label": "", "type": "a" }, { "href": "https://huggingface.co/pcuenq", "label": "", "type": "a" }, { "href": "https://huggingface.co/merve", "label": "", "type": "a" }, { "href": "https://huggingface.co/IlyasMoutawwakil", "label": "", "type": "a" }, { "href": "https://huggingface.co/ArthurZ", "label": "", "type": "a" }, { "href": "https://huggingface.co/sergiopaniego", "label": "", "type": "a" }, { "href": "https://huggingface.co/Molbap", "label": "", "type": "a" }, { "href": "https://nlp.fast.ai/classification/2018/05/15/introducing-ulmfit.html", "label": "ULMFiT", "type": "a" }, { "href": "https://huggingface.co/papers/2001.08361", "label": "Scaling laws", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/moe_routing.png", "label": "", "type": "a" }, { "href": "https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mixture-of-experts", "label": "Maarten Grootendorst", "type": "a" }, { "href": "https://huggingface.co/openai/gpt-oss-20b", "label": "gpt-oss-20b", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/gpt-oss-20-inference.mp4", "label": "Your browser does not support the video tag.", "type": "source" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/faster_training.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/papers/2409.02060", "label": "OLMoE: Open Mixture-of-Experts Language Models", "type": "a" }, { "href": "https://huggingface.co/collections/Qwen/qwen35", "label": "Qwen 3.5", "type": "a" }, { "href": "https://huggingface.co/collections/MiniMaxAI/minimax-m2", "label": "MiniMax M2", "type": "a" }, { "href": "https://huggingface.co/collections/zai-org/glm-5", "label": "GLM-5", "type": "a" }, { "href": "https://huggingface.co/collections/moonshotai/kimi-k25", "label": "Kimi K2.5", "type": "a" }, { "href": "https://huggingface.co/deepseek-ai/DeepSeek-R1", "label": "DeepSeek R1", "type": "a" }, { "href": "https://huggingface.co/deepseek-ai/DeepSeek-V2", "label": "DeepSeek V2", "type": "a" }, { "href": "https://huggingface.co/mistralai/Mixtral-8x7B-v0.1", "label": "Mixtral-8x7B", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/moe_2y_timeline.png", "label": "", "type": "a" }, { "href": "https://x.com/soumithchintala/status/1671267150101721090", "label": "rumored", "type": "a" }, { "href": "https://huggingface.co/collections/openai/gpt-oss", "label": "gpt-oss models", "type": "a" }, { "href": "https://huggingface.co/blog/moe", "label": "this blog", "type": "a" }, { "href": "https://youtu.be/CDnkFbW-uEQ", "label": "YouTube video on routing", "type": "a" }, { "href": "https://huggingface.co/docs/transformers/main/en/model_doc/auto", "label": "AutoModelForCausalLM.from_pretrained(\"model_id\")", "type": "a" }, { "href": "https://huggingface.co/deepseek-ai/DeepSeek-V3/raw/main/model.safetensors.index.json", "label": "DeepSeek-V3 checkpoint index", "type": "a" }, { "href": "https://huggingface.co/kernels-community/megablocks", "label": "grouped GEMMs and fused MoE implementations", "type": "a" }, { "href": "https://github.com/huggingface/transformers/pull/41580", "label": "weight loading refactor", "type": "a" }, { "href": "https://huggingface.co/docs/transformers/main/en/weightconverter", "label": "generic WeightConverter", "type": "a" }, { "href": "https://github.com/huggingface/transformers/blob/main/src/transformers/core_model_loading.py", "label": "MergeModulelist", "type": "a" }, { "href": "https://github.com/huggingface/transformers/blob/b71de73468429eb02da18caa50e9b5200400a4ed/src/transformers/core_model_loading.py", "label": "SplitModulelist", "type": "a" }, { "href": "https://github.com/ariG23498/transformers/tree/bench-v4", "label": "https://github.com/ariG23498/transformers/tree/bench-v4", "type": "a" }, { "href": "https://github.com/ariG23498/transformers/tree/bench-v5", "label": "https://github.com/ariG23498/transformers/tree/bench-v5", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/loading_benchmark.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/docs/transformers/experts_interface", "label": "Experts Backend system", "type": "a" }, { "href": "https://github.com/huggingface/transformers/pull/42697", "label": "PR #42697", "type": "a" }, { "href": "https://docs.pytorch.org/docs/stable/generated/torch.bmm.html", "label": "torch.bmm", "type": "a" }, { "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.grouped_mm.html", "label": "torch._grouped_mm", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/moe-transformers/expert_backend.png", "label": "", "type": "a" }, { "href": "https://github.com/huggingface/transformers/blob/b71de73468429eb02da18caa50e9b5200400a4ed/src/transformers/integrations/tensor_parallel.py", "label": "GroupedGemmParallel", "type": "a" }, { "href": "https://unsloth.ai/docs/new/faster-moe", "label": "Unsloth’s official guide", "type": "a" }, { "href": "https://huggingface.co/blog/native-speed-vllm-transformers-backend", "label": "vllm transformers inference Native-speed vLLM transformers modeling backend 77 July 8, 2026", "type": "a" }, { "href": "https://huggingface.co/blog/continuous_async", "label": "transformers pytorch optimization Unlocking asynchronicity in continuous batching 68 May 14, 2026", "type": "a" }, { "href": "https://huggingface.co/bambuuai", "label": "bambuuai", "type": "a" }, { "href": "https://huggingface.co/join?next=%2Fblog%2Fmoe-transformers", "label": "Sign up", "type": "a" }, { "href": "https://huggingface.co/yantardev", "label": "", "type": "a" }, { "href": "https://huggingface.co/hunkycolony", "label": "", "type": "a" }, { "href": "https://huggingface.co/lewtun", "label": "", "type": "a" }, { "href": "https://huggingface.co/AINovice2005", "label": "", "type": "a" }, { "href": "https://huggingface.co/burtenshaw", "label": "", "type": "a" }, { "href": "https://huggingface.co/regisss", "label": "", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:53:22Z", "source_url": "https://huggingface.co/blog/moe-transformers", "papers": [], "actual_video_files": [], "body_chars": 14646, "expected_body_images": 25, "discovery_status": "ok", "delivery_status": "partial", "missing_reason": "invalid_or_wrong_resolution", "raw_html": "attachments/e44d72287f61/pages/source.html", "local_html": "attachments/e44d72287f61/pages/article.html", "body_base_path": "attachments/e44d72287f61/pages/", "body_markdown_path": "attachments/e44d72287f61/pages/article.md" }