Plain-English Summary
DeepSeek-V2 demonstrated that architectural innovation could dramatically reduce inference costs while maintaining frontier-level performance. The model has 236 billion total parameters but activates only 21 billion per token through a fine-grained mixture-of-experts architecture with 160 experts (routing to 6 per token). Combined with Multi-head Latent Attention (MLA), which compresses key-value cache by 93%, the model achieves 5-7x cheaper inference than comparable dense models.
DeepSeek priced API access at roughly 1/30th the cost of GPT-4, demonstrating that Chinese labs could compete on both capability and cost.
Key Innovation
Multi-head Latent Attention (MLA) was the key architectural contribution. Standard multi-head attention requires caching keys and values for all attention heads, which dominates memory during inference with long sequences. MLA compresses the key-value representation into a low-rank latent space, dramatically reducing the KV-cache size without sacrificing attention quality.
Combined with the fine-grained MoE architecture (many small experts rather than few large ones), this created a model optimized end-to-end for inference efficiency.
Impact on the Field
DeepSeek-V2 signaled that Chinese AI labs could innovate at the architectural level rather than merely replicating Western approaches at scale. The paper’s efficiency innovations have been widely studied and partially adopted across the industry. The aggressive pricing forced Western labs to reconsider their cost structures.
The model also demonstrated that mixture-of-experts and attention compression were complementary efficiency techniques that could be combined for multiplicative gains.
Models That Built on This
DeepSeek-V3 extended the architecture to larger scale with improved training techniques. DeepSeek-R1 applied the architecture to reasoning with reinforcement learning. The MLA technique has influenced attention design in subsequent models from multiple labs. DeepSeek’s efficiency innovations contributed to the broader narrative that frontier AI does not require frontier capital expenditure.