Model distillation transfers knowledge from a large, expensive “teacher” model to a smaller, faster “student” model. Rather than training the student on raw data, the student learns from the teacher’s outputs — its probability distributions over tokens, its reasoning patterns, and its intermediate representations. Because the teacher’s outputs contain richer learning signals than raw labels (the phenomenon Geoffrey Hinton called “dark knowledge” in his seminal 2015 paper), the student can achieve 90-99% of the teacher’s performance with 10-100x fewer parameters, making it dramatically cheaper and faster to serve at inference time.
Distillation has become the defining production technique of the current AI era. It is the reason capable AI models can run on phones, the reason API pricing spans a 100x range from frontier to budget models, and the reason open-source models have closed much of the gap with proprietary systems. Understanding distillation is essential to understanding the economics of deploying AI at scale.
The mechanics of distillation
Classical distillation, as introduced by Hinton, Valilov, and Dean in 2015, works by training the student model to match the teacher’s full output distribution rather than just the hard labels. The insight is that the teacher’s probability distribution contains information that hard labels do not.
Consider a teacher model classifying an image. The hard label says “cat.” But the teacher’s output distribution might assign 0.7 probability to “cat,” 0.15 to “lynx,” 0.08 to “dog,” and 0.02 to “fox.” These secondary probabilities — the dark knowledge — encode the teacher’s understanding of inter-class relationships. The student that learns “this image is cat-like but somewhat lynx-like” acquires richer knowledge than one that only learns “this is a cat.”
For large language models, distillation typically follows this process:
-
Data generation. The teacher model generates millions of input-output examples across the target task distribution. For a general-purpose language model, this means diverse prompts spanning coding, reasoning, conversation, analysis, and creative tasks. The quality and diversity of this dataset largely determine the student’s capabilities.
-
Soft target training. The student is trained to match the teacher’s token-level probability distributions, using a loss function that combines the standard cross-entropy loss with a KL-divergence term measuring how well the student’s output distribution matches the teacher’s. A “temperature” parameter softens the distributions, making the dark knowledge more accessible.
-
Task-specific refinement. After general distillation, the student may be further fine-tuned on specific tasks where the teacher excels but the student lags. This targeted approach helps close capability gaps on the hardest tasks without expensive full-distribution matching.
What gets distilled: a taxonomy
Not all distillation is the same. The field has developed distinct approaches depending on what knowledge is transferred:
Output distillation trains the student on the teacher’s final outputs — the generated text or token probabilities. This is the simplest and most common approach. OpenAI’s GPT-4o mini, Meta’s Llama 3.2 1B and 3B models, and Google’s Gemma family all use variants of output distillation. It reliably transfers the teacher’s general capabilities but may miss reasoning processes that are not reflected in final outputs.
Reasoning distillation trains the student on the teacher’s chain-of-thought traces — the step-by-step reasoning the teacher uses to arrive at answers. This is particularly important for math, logic, and coding tasks where the reasoning process matters as much as the answer. DeepSeek’s R1 distilled models and OpenAI’s o1-mini demonstrate that smaller models can acquire sophisticated reasoning capabilities through this approach. DeepSeek-R1-Distill-Qwen-32B, a 32B model distilled from the 671B DeepSeek-R1, achieves 94.3% of the teacher’s performance on AIME 2024 math benchmarks.
Feature distillation trains the student to match the teacher’s internal representations — the activations in intermediate layers. This transfers the teacher’s learned feature space, potentially capturing abstract concepts that are not visible in outputs. Feature distillation requires architectural compatibility between teacher and student, making it less flexible but sometimes more effective for closely related model sizes.
Behavioral distillation focuses not on matching distributions but on matching the teacher’s behavior in specific scenarios — how it handles ambiguity, when it refuses requests, how it formats outputs. This is particularly relevant for alignment, where the student needs to inherit the teacher’s safety behaviors. Anthropic’s approach to training the Haiku model series incorporates behavioral distillation to maintain safety properties across model sizes.
Distillation in production: who distills what
| Model family | Teacher | Student | Parameter ratio | Performance retention | Key use case |
|---|---|---|---|---|---|
| GPT-4o / GPT-4o mini | GPT-4o | GPT-4o mini | ~10-20x | ~92% on MMLU | High-throughput, cost-sensitive APIs |
| Gemini / Gemma | Gemini Ultra/Pro | Gemma 2 2B/9B/27B | ~50-100x | ~85-93% on standard benchmarks | On-device, open-source deployment |
| Claude Opus / Haiku | Claude Opus 4 | Claude Haiku 3.5 | ~15-20x (est.) | ~88% on general tasks | Low-latency, high-volume workloads |
| Llama 3.1 / 3.2 | Llama 3.1 405B | Llama 3.2 1B/3B | 135-405x | ~78-85% on core benchmarks | Edge devices, mobile, embedded |
| DeepSeek-R1 / R1-Distill | DeepSeek-R1 671B | R1-Distill 1.5B-70B | 10-447x | ~80-94% on math/reasoning | Reasoning at lower cost |
| Phi-4 | Proprietary large models | Phi-4 14B | ~30-50x (est.) | ~90% on coding/reasoning | Efficient desktop/laptop inference |
The economics of distillation
Distillation transforms the cost curve of AI deployment. The numbers are stark:
Inference cost. Claude Opus 4 costs $15 per million input tokens and $75 per million output tokens. Claude Haiku 3.5, distilled to a fraction of the size, costs $0.80/$4 — roughly 19x cheaper on input and 19x cheaper on output. For an enterprise processing 100 million tokens per day, this translates from ~$4,500/day to ~$240/day: a difference of $1.5 million annually.
Latency. Smaller models generate tokens faster. Haiku-class models typically achieve 100-200 tokens per second versus 30-60 for frontier models. For interactive applications — chatbots, autocomplete, real-time search — this latency difference determines user experience.
Hardware requirements. A 405B-parameter model in FP16 requires ~810 GB of GPU memory — at least 10 A100-80GB GPUs just for inference. A distilled 8B model fits on a single consumer GPU. This difference determines whether AI runs in expensive cloud data centers or on-premises, on laptops, or on phones.
Distillation cost. The process itself is expensive. Generating millions of teacher outputs can cost $50,000-500,000 in API calls, and student training requires substantial GPU resources. But this is a one-time cost amortized across every inference call the student model ever handles — often billions of requests. The ROI is typically realized within days of deployment.
Distillation vs. other compression techniques
Distillation is one of several approaches to making models smaller and cheaper. Understanding how they relate:
| Technique | What it does | Quality impact | Speed improvement | When to use |
|---|---|---|---|---|
| Distillation | Trains a new smaller model from a larger one | 1-15% degradation | 5-20x | Building a new production model |
| Quantization | Reduces numerical precision of existing weights | 0.5-5% degradation | 1.5-4x | Deploying an existing model more cheaply |
| Pruning | Removes less important weights/neurons | 2-10% degradation | 1.5-3x | Optimizing a specific model architecture |
| Distillation + Quantization | Combined approach | 2-18% degradation | 10-50x | Maximum efficiency for edge deployment |
These techniques are complementary, not competing. A common production pipeline distills a frontier model to a smaller architecture, then quantizes the distilled model for deployment. Meta’s Llama 3.2 1B is first distilled from Llama 3.1 405B, then quantized to 4-bit for mobile deployment, achieving roughly 200x reduction in compute requirements with 75-80% quality retention.
The distillation debate: legal and ethical dimensions
Distillation has become one of the most contested topics in AI policy, centered on a fundamental question: who owns the knowledge encoded in a model’s outputs?
The licensing landscape. OpenAI’s terms of service prohibit using its API outputs to train competing models. Google’s Gemini API terms include similar restrictions. Anthropic’s usage policy restricts using Claude outputs to develop competing AI models. These restrictions are designed to prevent competitors from distilling proprietary models at a fraction of the training cost.
The enforcement challenge. Detecting whether a model was distilled from a proprietary teacher is technically difficult. Model outputs are text — and text generated by different models on similar prompts will naturally share characteristics. Research into “model fingerprinting” attempts to identify telltale patterns that indicate distillation, but the field is nascent and results are contested.
The open-source argument. Advocates for open-source AI argue that distillation restrictions concentrate AI capabilities in a few large companies. They point out that Meta, Google, and Microsoft have all released models that benefit from distillation, and that restricting the practice limits competition and innovation. The counter-argument is that training frontier models costs $100M+ and companies need to recoup that investment.
The synthetic data distinction. An emerging legal and technical question is where distillation ends and synthetic data generation begins. If a company uses a frontier model to generate training data on topics the model has general knowledge about (not copying specific outputs), is that distillation? Courts and regulators have not yet provided clear answers.
Advanced distillation techniques
The field has moved well beyond basic output matching:
Progressive distillation uses a chain of intermediate-sized models rather than distilling directly from the largest to the smallest. A 405B teacher first distills to a 70B student, which then distills to a 13B student, which distills to a 3B student. Each step preserves more information than a single large jump. Google’s Gemma model family uses this approach.
Self-distillation involves a model distilling from its own outputs after filtering for quality. The model generates many solutions to a problem, the correct ones are identified (through verification, unit tests, or human evaluation), and the model is trained on its own best outputs. This is a core technique behind reasoning model improvements — DeepSeek, OpenAI, and Google all use variants.
Multi-teacher distillation trains a student from multiple teachers simultaneously, potentially combining strengths. A student might learn coding from one teacher, reasoning from another, and conversational ability from a third. This requires careful balancing to avoid conflicting learning signals.
Online distillation generates teacher outputs on-the-fly during student training rather than pre-generating a static dataset. This is more expensive but produces higher-quality students because the training distribution adapts to the student’s weaknesses.
Limitations and risks of distillation
Distillation is powerful but not without pitfalls:
Capability cliffs. Distilled models sometimes exhibit sharp capability drops on tasks near the edge of the teacher’s abilities. A teacher that scores 85% on a difficult reasoning benchmark might produce a student that scores 75% — but the student’s failures are often concentrated on the hardest 15% of problems, making it unreliable precisely where you need it most.
Alignment tax. Safety-aligned behaviors can be lost during distillation if they are not explicitly represented in the training data. A teacher that reliably refuses harmful requests might produce a student that sometimes complies, because refusal behavior is underrepresented relative to helpful responses. This is why labs like Anthropic invest specifically in safety-focused distillation.
Evaluation gaming. Because distillation datasets often include benchmark-like examples, distilled models can overfit to standard evaluations. A distilled model might score well on MMLU or HumanEval while performing worse than expected on real-world tasks not represented in the training distribution.
Diminishing returns. Beyond a certain compression ratio, distillation quality degrades rapidly. Distilling a 400B model to 70B retains most capabilities. Distilling to 7B loses meaningfully more. Distilling to 1B produces a model that may lack fundamental reasoning capabilities that no amount of teacher knowledge can compensate for, because the student simply does not have enough parameters to represent complex concepts.
Frequently asked questions
What is the difference between distillation and fine-tuning? Fine-tuning trains a model on task-specific data to specialize its behavior. Distillation trains a smaller model to replicate a larger model’s general capabilities. Fine-tuning does not change model size; distillation typically produces a smaller model. In practice, distillation can be viewed as a special form of fine-tuning where the training data comes from a teacher model rather than from human annotations or raw data.
Can you distill open-source models? Yes, and this is common practice. Meta’s Llama models, Mistral’s models, and others with permissive licenses can be freely distilled. The legal restrictions apply primarily to proprietary API-based models (OpenAI, Anthropic, Google) whose terms of service prohibit using outputs for competitive model training. Open-source distillation is a major driver of the broader AI ecosystem.
How much data do you need for distillation? It depends on the task breadth and compression ratio. Distilling a general-purpose model typically requires 1-10 million teacher-generated examples across diverse tasks. Task-specific distillation can work with 10,000-100,000 examples. The quality and diversity of examples matter more than raw volume — carefully curated datasets often outperform larger but noisier ones.
Does distillation always result in a worse model? Usually, but not always on every task. Distilled models sometimes outperform their teachers on specific narrow tasks because the distillation process can act as a form of regularization, reducing overfitting. However, on broad capability benchmarks, the teacher is almost always superior. The value proposition is not that distillation produces a better model but that it produces a dramatically cheaper model that is good enough for the target use case.
Will distillation become less important as models get cheaper? Unlikely. Even as frontier model inference costs decline, the demand for AI on edge devices, mobile phones, and embedded systems grows faster. A phone cannot run a 400B model regardless of price. And enterprise deployments processing billions of requests per day will always benefit from cost reduction. Distillation is a fundamental technique that adapts to whatever the current frontier is — as frontier models get larger and more capable, the gap between frontier and edge creates ongoing demand for distillation.