Model Distillation: How Enterprises Get Frontier Quality at Small-Model Cost

Model distillation is the technique of training a small, cheap model to reproduce the outputs of a large, expensive one for a specific task. A frontier "teacher" model generates thousands of high-quality input-output examples for a narrowly defined job, the examples are filtered for quality, and a small "student" model is fine-tuned on them until its performance on that job approaches the teacher's. The economics are the reason executives should care: on published price lists as of mid-2026, frontier models charge on the order of ten to seventy five USD per million output tokens while small-tier models charge well under one USD per million, a spread of ten to seventy five times for workloads that may not need frontier capability at all. Distillation is how an organization closes the quality gap that normally justifies paying the frontier premium, and it is the technique most enterprise AI teams have heard of but never actually run.

Key Takeaways

• Distillation transfers capability for one narrow task, not general intelligence. The student matches the teacher on the distribution it was trained on and degrades sharply outside it, which is precisely why it works: narrow tasks do not need general intelligence.
• It wins on high-volume, narrow, stable tasks such as classification, extraction, routing, summarization in a fixed format, and structured rewriting. It fails on open-ended reasoning, fast-moving requirements, and tasks where the input distribution is broad or unpredictable.
• Distillation differs from ordinary fine-tuning in where the labels come from. Fine-tuning on raw data requires labeled examples an organization usually does not have; distillation manufactures those labels with a teacher model, which is why it scales.
• The provider-terms question is real. Major API providers prohibit using their outputs to train competing models, while several open-weight licenses explicitly permit distillation. The compliant paths are the provider's own distillation tooling, distillation into a model on the same platform, or an open-weight teacher.
• The economics are a one-time project cost against a recurring per-token saving, so the decision reduces to a break-even calculation driven almost entirely by volume. Below roughly a million requests a month the project rarely pays; well above it, the payback is often measured in months.

What Distillation Actually Is, and What It Is Not

The idea is older than the current model generation. The canonical early result is DistilBERT (Sanh et al., 2019), which compressed BERT into a model with 40 percent fewer parameters that ran 60 percent faster while retaining about 97 percent of its language-understanding performance. The current generation of frontier providers uses the same principle at scale: Google has described Gemini Flash models as distilled from their larger Pro siblings, and DeepSeek's R1 release in January 2025 shipped a family of small open-weight students distilled from its flagship reasoning model. The technique is not exotic. It is how the model market already manufactures its own cheap tier.

For an enterprise, the practical form is sequence-level distillation: the teacher produces complete, high-quality outputs for real task inputs, and the student is fine-tuned on those input-output pairs. This is different from the research variant that matches the teacher's internal probability distributions, which requires access to model internals that API customers do not have. It is also different from two things it is commonly confused with.

Distillation is not fine-tuning on raw data. Ordinary fine-tuning requires a labeled dataset, and the labeled dataset is the expensive part: someone has to produce thousands of correct outputs. Distillation solves the labeling problem by making the teacher model do it. The organization supplies representative inputs, which it usually has in abundance from logs and historical records, and the teacher supplies the outputs. The quality of the student is therefore bounded by the quality of the teacher plus the quality of the filtering step, not by the organization's willingness to pay human annotators.

Distillation is not prompt migration. Teams sometimes swap a frontier model for a small off-the-shelf model, keep the same prompt, watch quality drop, and conclude that small models cannot do the job. That experiment tests the small model's zero-shot ability, not its trainable ceiling. A small model that scores poorly out of the box on a narrow task will frequently match the frontier model on that same task after distillation, because the task-specific behavior gets baked into its weights instead of being carried in the prompt. The decision between prompting a small model and distilling into one is covered in more depth in the guide to when small language models are enough for enterprise work, and the broader choice of adaptation technique is mapped in fine-tuning vs RAG vs long context.

The Distillation Pipeline: Five Stages

The pipeline is straightforward to describe and unforgiving to shortcut. Organizations that fail at distillation almost always failed at stage one or stage three, not at model training.

Stage one: task definition. Distillation only works when the task is narrow enough to enumerate. "Answer customer questions" is not a distillable task; "classify inbound support tickets into one of fourteen categories and extract the account identifier" is. The definition must include the output format, the input distribution, and the evaluation criteria before any generation begins, because every later stage inherits its assumptions.

Stage two: teacher generation. The frontier model generates outputs for a large sample of real inputs, typically in the range of ten thousand to a hundred thousand examples depending on task complexity. This is the stage where the frontier premium is paid on purpose: paying frontier rates once to create a training set is the whole trade. Prompting matters here; the teacher should run with the best prompt, examples, and context the team can build, because the student will learn whatever the teacher produced, including its mistakes.

Stage three: filtering and quality control. Raw teacher output is not training data. Generations get scored, deduplicated, and filtered: outputs that violate the format, contradict the input, or fall below a quality threshold are dropped, often using the teacher itself or a second model as a grader. Skipping this stage is the most common cause of a disappointing student. A student trained on unfiltered teacher output learns the teacher's failure modes with the same fidelity as its successes.

Stage four: student training. The filtered pairs fine-tune a small model. The student can be a small proprietary model on the same provider platform, or an open-weight model in the single-digit-billions parameter range served on the organization's own infrastructure. The training run itself is usually the cheapest and fastest stage of the pipeline.

Stage five: evaluation against the teacher. The student is scored side by side with the teacher on a held-out set the student never saw. The relevant question is not whether the student matches the teacher everywhere; it will not. The question is whether the gap on the defined task is inside the tolerance the business set at stage one. This evaluation harness is not a one-time gate: it becomes the permanent regression suite that detects drift when the input distribution shifts. Building that harness properly is its own discipline, covered in the guide to enterprise LLM evaluation harnesses.

When Distillation Wins and When It Fails

The pattern in the tasks where distillation succeeds is consistent: high volume, narrow scope, stable requirements. Classification, entity and field extraction, routing and triage, format conversion, fixed-template summarization, and policy-constrained rewriting are the canonical wins. These tasks share three properties. The output space is constrained, so the student has less to learn. The input distribution is observable in advance from logs, so the training set can cover it. And the requirements change slowly, so the one-time training cost amortizes over a long service life.

The failure cases are the mirror image. Open-ended reasoning does not distill well: multi-step problems whose solution path varies per input require capability, not learned behavior, and a small student cannot absorb capability it does not have room for. Fast-moving requirements destroy the amortization: if the task definition changes quarterly, each change means regenerating data and retraining, and the project cost recurs instead of amortizing. Broad input distributions defeat coverage: a student is only as good as the distribution it saw, and a task whose inputs are genuinely unpredictable will keep surfacing inputs the training set never covered. The evidence from teams running distilled models in production is that the failure is rarely dramatic; it is a slow accumulation of edge-case errors on inputs the training set underrepresented, which is why the stage-five evaluation harness has to stay in place for the life of the model.

There is one more failure mode worth naming because it is organizational rather than technical: distilling before establishing that the frontier model actually solves the task. The teacher's quality is the student's ceiling. A team that cannot get acceptable output from the frontier model with good prompting has a task-definition problem, and distillation will faithfully reproduce it at lower cost.

The Provider-Terms Question

Training a student on a teacher's outputs raises a contractual question that enterprise legal teams should settle before the pipeline runs, not after. The major API providers, including OpenAI, Anthropic, and Google, have usage terms that prohibit using their model outputs to develop or train models that compete with theirs. What counts as "competing" is the operative ambiguity, and the safe reading is conservative: distilling a frontier API's outputs into a model you serve yourself, or into another provider's model, sits somewhere between gray and prohibited depending on the terms in force.

Three paths avoid the problem cleanly. First, the provider's own distillation tooling. OpenAI launched a first-party model distillation product in October 2024 that stores frontier-model completions and fine-tunes its own smaller models on them, keeping the whole loop inside one platform; distilling into a smaller model on the same provider's platform is the pattern the providers themselves commercialize. Second, an open-weight teacher. Meta's Llama license from version 3.1 onward explicitly permits training on Llama outputs, with the requirement that derivative model names carry a Llama prefix, and DeepSeek's R1 shipped under an MIT license alongside distilled students, making the permission unambiguous. A strong open-weight teacher costs more to operate per token than a small API model but removes the contractual constraint entirely. Third, negotiated enterprise terms. Organizations with enterprise agreements can and do negotiate distillation rights for internal, non-competing use; the ask is more routine than most procurement teams assume.

The one path that does not hold up is the quiet one: running the pipeline against a frontier API and assuming nobody will notice. Providers log usage patterns, bulk generation against a fixed task template is a recognizable signature, and the downside is not a fee but a terminated account under an agreement the rest of the AI program depends on.

The Economics: One-Time Cost Against Per-Token Savings

The financial structure of a distillation project is a one-time cost against a recurring saving, which means the whole decision is a break-even calculation and the dominant variable is volume.

The one-time cost has four components: engineering time to define the task and build the pipeline, teacher-generation spend at frontier rates, training compute, and the evaluation harness. For a well-scoped task, the evidence from published case studies and vendor tooling puts the typical all-in range in the low tens of thousands of USD, with teacher generation usually the smallest line item and engineering time the largest. The recurring saving is the per-token spread between frontier and small-tier rates multiplied by volume, minus the cost of serving or renting the student.

A worked illustration makes the shape clear. Take a document-classification task running fifty million requests a year at roughly eight hundred tokens per request, which is forty billion tokens annually. At a blended frontier rate of five USD per million tokens, that is two hundred thousand USD a year. The same volume on a distilled small model at twenty five cents per million tokens is ten thousand USD a year. If the distillation project costs fifty thousand USD all-in, it pays back in about three months and returns roughly four times its cost in year one. Run the same numbers at five hundred thousand requests a year and the annual saving is under two thousand USD, which never repays the project. The figures are illustrative, but the structure is not: volume is the decision. Note also that per-token list price understates true frontier spend once retries, long prompts, and orchestration overhead are counted, a gap examined in the hidden costs behind LLM token pricing, which means real break-evens tend to arrive sooner than the list-price math suggests.

The decision has a third option that the frontier-versus-distilled framing hides: a small off-the-shelf model with good prompting and no training at all. It is the right answer more often than teams expect, and it costs a week of evaluation to rule in or out.

How to Decide: A Four-Question Sequence

The decision compresses into four questions asked in order. First, is the task narrow, high-volume, and stable? If not, stop; stay on the frontier API. Second, does a small off-the-shelf model already pass the evaluation with good prompting? If yes, migrate the prompt and skip training entirely. Third, does the break-even arrive inside twelve months at realistic volume? If not, the engineering time is better spent elsewhere. Fourth, is there a clean contractual path for the teacher? If all four answers hold, distillation is not a research project; it is a cost-reduction project with a defined payback, and the organizations already running it are quietly serving frontier-quality output on the cheapest tier of the market.

FAQ

What is model distillation in simple terms?

Model distillation trains a small, inexpensive model to imitate a large, expensive one on a specific task. The large model, called the teacher, generates thousands of example outputs for real inputs; the examples are filtered for quality; and the small model, called the student, is fine-tuned on them. The result is a model that performs close to the teacher on that one task at a fraction of the per-token cost.

How is distillation different from fine-tuning?

Fine-tuning is the training mechanism; distillation is a strategy for where the training data comes from. Ordinary fine-tuning needs a labeled dataset that humans produce, which is slow and expensive. Distillation manufactures the labels by having a teacher model generate them, then fine-tunes the student on the result. Every distillation project includes fine-tuning, but most fine-tuning projects are not distillation.

Is it legal to distill from OpenAI, Anthropic, or Google models?

The major providers' usage terms prohibit using their outputs to train competing models, so distilling a frontier API into a model you serve independently is contractually risky. The clean paths are the provider's own distillation tooling, which keeps the loop on one platform; an open-weight teacher such as Llama or DeepSeek whose license permits training on outputs; or negotiated enterprise terms. Legal review belongs before the pipeline runs, not after.

How much does a distillation project cost?

For a well-scoped narrow task, typical all-in cost lands in the low tens of thousands of USD, dominated by engineering time rather than compute or teacher-generation spend. The recurring saving is the frontier-to-small price spread times annual token volume. As a rule of thumb, tasks below roughly a million requests a month rarely repay the project; tasks well above it often pay back within a quarter.

When should an enterprise not use distillation?

Skip distillation for open-ended reasoning tasks, for tasks whose requirements change quarterly or faster, for low-volume workloads, and for any task where a small off-the-shelf model already passes evaluation with good prompting. Distillation is a cost-reduction technique for narrow, stable, high-volume work; it does not make a small model generally smarter.

Related reading on FinTekCafe

• Small Language Models in the Enterprise: When Smaller Is the Right Call
• Fine-Tuning vs RAG vs Long Context: The Decision Framework
• The Hidden Cost of LLM Inference: Why Token Pricing Is Misleading
• The LLM Evaluation Harness: The Enterprise Guide