Mixture of Experts (MoE) is a sparse neural network architecture where the model contains many parallel sub-networks — called “experts” — but only activates a small subset of them for each input token. A gating network (the “router”) examines each token and decides which experts should process it. The result is a model that stores far more knowledge than it uses on any given input: a 400B-parameter MoE model might activate only 50B parameters per token, achieving the quality of a much larger dense model at the computational cost of a much smaller one.
This fundamental decoupling of total model capacity from per-token compute cost is why MoE has become the dominant architecture for scaling frontier language models. When you need a model that knows everything about medicine, law, coding, mathematics, creative writing, and 100 other domains, MoE lets you have specialized capacity for each domain without paying the full compute cost of a model that large on every single token.
How mixture of experts works
A standard dense transformer model processes every token through every parameter in every layer. In an MoE transformer, selected layers (typically the feed-forward network layers, which constitute about two-thirds of model parameters) are replaced with MoE layers. Each MoE layer contains:
N expert networks. Each expert is a feed-forward network (FFN) with its own weights. In a typical configuration, there might be 8, 16, 64, or even 128 experts per layer. Each expert has the same architecture but learns different specializations through training.
A router (gating network). A small neural network that takes each token’s hidden state as input and produces a probability distribution over experts. The router selects the top-k experts (usually k=1 or k=2) to process that token. The router’s weights are learned jointly with the experts during training.
A combining function. The outputs of the selected experts are combined using the router’s probability weights — typically a weighted sum. If two experts are selected with weights 0.7 and 0.3, the final output is 0.7 times expert A’s output plus 0.3 times expert B’s output.
The computation flow for each token through an MoE layer:
- The token’s hidden state enters the router
- The router produces scores for all N experts
- The top-k experts are selected based on scores
- Only the selected experts perform their forward pass on the token
- Expert outputs are weighted by router scores and summed
Everything outside the MoE layers — the attention layers, embeddings, and layer norms — remains dense and is shared across all tokens. This means MoE models are “sparse” only in their FFN layers, while their attention mechanism processes every token identically.
The MoE efficiency equation
The power of MoE comes from a simple mathematical relationship. Consider two models:
- Dense model A: 200B total parameters, all active per token
- MoE model B: 800B total parameters, 8 experts with top-2 routing = ~200B active per token
Both models use approximately the same compute per token (200B active parameters), so inference speed is similar. But MoE model B has 4x the total capacity — 4x the stored knowledge, 4x the specialization potential. In practice, the MoE model performs significantly better because different experts can specialize in different domains, effectively giving the model a much richer knowledge base.
The efficiency gains are real and measurable:
| Metric | Dense 70B | MoE ~13B active / 47B total (Mixtral 8x7B) |
|---|---|---|
| Active parameters per token | 70B | ~13B |
| Total parameters | 70B | 46.7B |
| MMLU accuracy | ~70% | ~71% |
| Inference speed (tokens/sec) | 1x | ~3-4x faster |
| Memory required (FP16) | 140 GB | 94 GB (all experts in memory) |
| Quality per FLOP | Baseline | ~2-3x better |
Mixtral 8x7B demonstrated that an MoE model with 13B active parameters could match a dense 70B model’s quality — achieving the same output quality at roughly one-fifth the compute cost per token.
The MoE model landscape
MoE has been adopted by virtually every organization building frontier models:
Production MoE models
| Model | Total params | Active params | Experts | Router config | Developer |
|---|---|---|---|---|---|
| Mixtral 8x7B | 46.7B | ~13B | 8 per layer | Top-2 | Mistral |
| Mixtral 8x22B | 141B | ~39B | 8 per layer | Top-2 | Mistral |
| GPT-4 (reported) | ~1.8T | ~280B (est.) | 16 per layer (est.) | Top-2 (est.) | OpenAI |
| Gemini 1.5 / 2.x | Undisclosed | Undisclosed | MoE confirmed | Undisclosed | |
| DeepSeek-V3 | 671B | 37B | 256 + 1 shared | Top-8 | DeepSeek |
| DeepSeek-R1 | 671B | 37B | 256 + 1 shared | Top-8 | DeepSeek |
| Llama 4 Scout | 109B | 17B | 16 experts | Top-1 | Meta |
| Llama 4 Maverick | 400B | 17B | 128 experts | Top-1 | Meta |
| DBRX | 132B | 36B | 16 per layer | Top-4 | Databricks |
| Grok-1 | 314B | ~86B | 8 per layer | Top-2 | xAI |
| Qwen3-235B | 235B | 22B | 128 per layer | Top-8 | Alibaba |
The DeepSeek architecture: pushing MoE boundaries
DeepSeek-V3 and its reasoning variant DeepSeek-R1 deserve detailed analysis because they pushed MoE design in novel directions that influenced the entire field:
Fine-grained experts. Instead of 8 or 16 large experts, DeepSeek-V3 uses 256 small experts plus 1 shared expert per MoE layer. The shared expert processes every token (providing a baseline representation), while the router selects 8 of the 256 specialized experts. Fine-grained experts enable more precise specialization — rather than having a “general science” expert, the model can have separate experts for organic chemistry, quantum mechanics, and molecular biology.
Auxiliary-loss-free load balancing. Previous MoE models used auxiliary losses during training to force balanced expert usage, which conflicted with the primary language modeling objective. DeepSeek introduced a bias-based balancing mechanism that adjusts expert selection probabilities without adding a competing loss term, producing better-balanced expert utilization with higher model quality.
Multi-token prediction. DeepSeek-V3 was trained with a multi-token prediction objective, predicting multiple future tokens simultaneously. This auxiliary objective provided richer training signal and enabled speculative decoding during inference — the model predicts several tokens at once, and a verification step confirms them.
The result: DeepSeek-V3 reportedly cost only $5.6 million to train, compared to estimated costs of $50-100 million for comparable dense models. Whether this number accounts for all R&D compute is debated, but the efficiency gain from MoE architecture was a significant factor.
Expert specialization: what do experts learn?
A natural question is whether MoE experts develop interpretable specializations — does expert 3 “know” science while expert 7 “knows” law? Research reveals a more nuanced picture:
Token-level routing, not topic-level. Experts specialize at the token level, not the document level. Within a single sentence about physics, different tokens may be routed to different experts. Function words (“the,” “is,” “of”) tend to route to a small set of common experts, while domain-specific content words route more variably.
Layer-dependent specialization. In deeper layers (closer to the output), experts tend to develop more semantic specialization. In early layers, routing patterns are more syntactic — based on word type and position rather than meaning.
Soft specialization with overlap. Experts do not have hard boundaries. An expert that predominantly handles scientific text will also process some literary text, legal text, and so on. Specialization is statistical, not categorical — an expert might process 15% of science tokens but only 5% of legal tokens, rather than 100% of one and 0% of the other.
Emergent redundancy. Multiple experts often develop overlapping capabilities, providing robustness. If one expert’s specialization is needed but it is at capacity, a second expert with partial overlap can handle overflow tokens adequately.
This nuanced specialization pattern is why simple interpretations (“expert 3 is the math expert”) are misleading. The reality is a complex, distributed specialization that resists clean human categorization.
The three core MoE challenges
1. Memory footprint
The central paradox of MoE is that while compute per token is low, all expert weights must be accessible. A model with 8 experts and 70B parameters per expert has 560B total parameters. Even with INT4 quantization, that requires 280 GB of memory — more than the VRAM of three H100 GPUs. Only 2 experts are active per token, but all 8 must be loaded or swappable.
Solutions include:
- Expert offloading: Keep inactive experts on CPU RAM or SSD and load them into GPU memory on demand. This works for low-batch-size inference but adds latency from data transfer.
- Expert parallelism: Distribute experts across multiple GPUs, with each GPU hosting a subset. Tokens are routed to the appropriate GPU via all-to-all communication. This is the standard approach for large-scale serving but requires high-bandwidth interconnects (NVLink, InfiniBand).
- Aggressive quantization: MoE models benefit more from quantization than dense models because the total parameter count is much larger relative to active parameters. INT4 or even INT2 quantization of expert weights can make MoE models fit on fewer GPUs with tolerable quality loss.
2. Load balancing
If the router sends most tokens to a few favored experts while others sit idle, the model wastes capacity and creates compute bottlenecks (the popular experts become the throughput constraint). Load imbalance manifests in several ways:
Expert collapse: Some experts receive so few tokens during training that they never develop useful specializations, effectively becoming dead parameters. This is the most severe form of imbalance.
Popularity skew: In practice, some experts are inherently more useful (e.g., an expert that handles common function words will be used more than one specializing in rare technical terms). Perfect balance is neither achievable nor desirable — the goal is preventing extreme imbalance.
Balancing techniques:
| Technique | How it works | Used by |
|---|---|---|
| Auxiliary balance loss | Adds a training loss term penalizing uneven expert usage | Mixtral, Switch Transformer |
| Expert capacity factor | Limits the maximum number of tokens per expert per batch, dropping overflow | Switch Transformer, GShard |
| Bias-based balancing | Adjusts router bias terms to steer traffic without auxiliary loss | DeepSeek-V3 |
| Expert choice routing | Experts select their top-k tokens rather than tokens selecting experts | Expert Choice (Zhou et al., 2022) |
| Random routing | Adds noise to router scores during training to prevent premature specialization | Early MoE work, ST-MoE |
3. Training instability
MoE models are harder to train stably than dense models. The router creates a discrete selection problem (which experts to activate), and the gradients through this discrete decision can be noisy. Common instability symptoms include:
- Router oscillation: The router rapidly shifts tokens between experts, preventing stable specialization
- Expert collapse cascades: One expert collapsing triggers others to absorb its traffic, creating further instability
- Loss spikes: Sudden increases in training loss, more frequent in MoE than dense models
Techniques to stabilize training include router z-loss (penalizing large router logits), careful learning rate schedules with warmup, and using a shared expert that processes all tokens (providing a stable baseline representation that MoE experts refine).
MoE vs. dense models: when to use which
| Consideration | Dense models | MoE models |
|---|---|---|
| Inference speed per token | Slower at equivalent quality | Faster (fewer active params) |
| Memory requirement | Proportional to quality | Higher (all experts loaded) |
| Training cost | Higher for equivalent quality | Lower (more capacity per FLOP) |
| Training stability | More stable | Requires careful balancing |
| Quantization effectiveness | Good | Excellent (more parameters to compress) |
| Hardware requirements | Standard GPU setups | Benefits from high-bandwidth interconnects |
| Fine-tuning complexity | Standard | More complex (which experts to update) |
| Batch inference throughput | Predictable | Variable (depends on routing patterns) |
| Single-GPU deployment | Straightforward | Often requires offloading or quantization |
Choose dense when: You need a small model for edge deployment, your hardware has limited memory, you need predictable latency, or you will extensively fine-tune the model.
Choose MoE when: You need maximum quality per FLOP, you have sufficient memory for all experts, you are serving at scale where throughput matters, or you need broad knowledge coverage across many domains.
MoE for fine-tuning and adaptation
Fine-tuning MoE models presents unique challenges and opportunities:
Full fine-tuning updates all expert weights. This is expensive and risks disrupting the learned routing patterns. In practice, full fine-tuning of large MoE models requires the same distributed infrastructure used for pre-training.
LoRA and parameter-efficient fine-tuning can be applied to MoE models by adding low-rank adapters to the active experts. This is the most practical approach for domain adaptation. A key design decision is whether to add adapters to all experts or only the router-selected ones, and whether to freeze the router during fine-tuning.
Expert-level adaptation is an emerging approach where new experts are added to handle a new domain, while existing experts are frozen. This is conceptually elegant — the model grows its capacity for the new domain without forgetting old knowledge — but requires modifying the router to incorporate the new experts.
MoE-ification converts a pre-trained dense model into an MoE model by duplicating the FFN layers into multiple experts and training a router. This can improve model quality without full pre-training, though results are mixed compared to MoE models trained from scratch.
The hardware story: why MoE demands specific infrastructure
MoE models create unique hardware requirements that differ from dense models:
Memory bandwidth over compute. Since fewer parameters are active per token but all must be accessible, MoE inference is even more memory-bandwidth-bound than dense inference. This makes high-bandwidth memory (HBM3/HBM3e) more critical than raw compute throughput.
All-to-all communication. In expert-parallel deployments, tokens must be routed to the GPU hosting the correct expert, processed, and returned. This all-to-all communication pattern requires high-bandwidth, low-latency interconnects. NVIDIA’s NVLink (900 GB/s per GPU on H100) and InfiniBand (400 Gb/s per port) are designed for this pattern. Deployments on commodity networking suffer severe communication bottlenecks.
GPU cluster topology matters. The physical arrangement of GPUs affects MoE serving efficiency. A model with 8 experts across 8 GPUs within a single node (connected by NVLink) performs very differently from the same model spread across 8 GPUs on different nodes (connected by network switches). Same model, same hardware count, vastly different latency.
This hardware sensitivity is why MoE models often underperform expectations when deployed on consumer or small-scale infrastructure. The architecture was designed for datacenter-class hardware with premium interconnects.
The future of MoE
Several research directions are actively evolving:
Extreme expert counts. DeepSeek-V3’s 256 experts was a breakthrough; research is exploring 1,000+ expert models where each expert is very small and highly specialized. This pushes toward a continuous spectrum between discrete experts and dynamic parameter generation.
Learned routing beyond top-k. Current routers make hard top-k decisions. Soft routing (every expert processes every token with varying weights) avoids load balancing issues but loses the compute efficiency. Research into dynamic-k routing (varying the number of active experts per token based on input complexity) aims to get the best of both.
MoE beyond FFN layers. Current MoE models only apply expert routing to feed-forward layers. Extending MoE to attention layers — with different attention experts capturing different relationship patterns — is an active research area with promising early results.
Modular and composable experts. Training experts that can be mixed, matched, and composed from different models. This would enable building custom models by combining a coding expert from one model with a medical expert from another — a kind of LEGO-block approach to model construction.
Frequently asked questions
Why do MoE models use more memory than dense models of similar quality? Because all expert weights must be loaded even though only a few are used per token. A model with 8 experts of 7B parameters each has 56B total parameters but only activates ~14B per token (with top-2 routing). The model performs like a 14B active-parameter model but requires memory for all 56B parameters. This is the fundamental MoE tradeoff: you get the quality of a much larger model at the inference speed of a smaller one, but pay the memory cost of the larger one.
Is GPT-4 a mixture of experts model? GPT-4’s architecture has not been officially confirmed by OpenAI, but multiple credible reports indicate it uses a MoE architecture with an estimated 1.8 trillion total parameters and approximately 16 experts per layer with top-2 routing, resulting in roughly 280 billion active parameters per token. This would explain how GPT-4 achieves its quality level while maintaining usable inference speeds — a dense 1.8T model would be prohibitively slow to serve.
Can I run MoE models on consumer hardware? Yes, with caveats. Mixtral 8x7B in INT4 quantization requires approximately 24 GB of VRAM, fitting on a single RTX 4090. Smaller MoE models like Qwen3-30B-A3B (30B total, 3B active) run comfortably on 8-16 GB GPUs. Larger MoE models (DeepSeek-V3 at 671B) require expert offloading to CPU RAM, which works but with significant speed penalties — expect 2-5 tokens per second versus 30-50 for a properly GPU-hosted deployment. Tools like llama.cpp and Ollama support MoE model loading with automatic expert management.
What is the difference between mixture of experts and model ensembling? Model ensembling runs the same input through multiple complete models and combines their outputs (typically by averaging or voting). Every model processes every input, so compute scales linearly with the number of models. MoE uses a single model with a learned router that activates only a subset of experts per token — compute stays roughly constant regardless of the total number of experts. Ensembling is a inference-time technique applied to independently trained models; MoE is a training-time architecture where experts are trained jointly and share representations through the router and non-expert layers.
Will MoE replace dense models entirely? Unlikely in the near term. Dense models remain preferred for small-scale deployment (edge devices, single GPUs), applications requiring predictable latency, extensive fine-tuning scenarios, and use cases where simplicity outweighs efficiency. MoE is dominant at the frontier scale where maximum quality per compute dollar matters. The industry appears to be converging on MoE for large general-purpose models and dense architectures for smaller, specialized models — though the boundary is shifting as MoE techniques improve.