Plain-English Summary
Language models generate text one token at a time, and each token requires a full forward pass through the model. For large models, this is slow. Speculative decoding accelerates this by using a small, fast “draft” model to propose several tokens at once, then having the large model verify them all in a single parallel forward pass. If the draft tokens are correct (which they often are for predictable text), you get multiple tokens for the cost of one large model call.
The key insight is that verification is cheaper than generation. The large model can check multiple proposed tokens simultaneously, while generating them one at a time would require sequential passes.
Key Innovation
The acceptance/rejection scheme is mathematically guaranteed to produce identical output distributions to the large model alone. This means speculative decoding provides a free speedup with zero quality degradation. The draft model proposes tokens, the target model computes probabilities for all proposed tokens in parallel, and a rejection sampling scheme determines how many to accept.
The speedup depends on the acceptance rate — how often the draft model’s predictions match the target model. For typical text, acceptance rates of 60-80% yield 2-3x speedups.
Impact on the Field
Speculative decoding became a standard inference optimization deployed by every major AI provider. The technique is particularly valuable for large models where the cost of a single forward pass is high, making the relative savings from parallelized verification substantial.
The approach also inspired related techniques: Medusa uses multiple prediction heads on a single model, lookahead decoding generates multiple futures simultaneously, and various draft model training strategies optimize the acceptance rate.
Models That Built on This
Google, OpenAI, and Anthropic all deploy speculative decoding in their inference infrastructure. The technique is integrated into inference frameworks like vLLM, TensorRT-LLM, and SGLang. It has become a standard component of production serving systems, typically providing 2-3x throughput improvements for conversational workloads.