Skip to content
Get in touch

Needs review · content version 2

Review Model Distillation

Your draft is bound to this exact source hash. If the term changes, the Review Desk will use a new storage key instead of silently attaching old decisions.

Definition

Model distillation is a training approach in which a smaller model learns to reproduce selected behavior of a larger or more capable teacher model.
Decision for Definition

Plain-Language Explanation

The smaller student model trains on outputs, probabilities, examples, or guidance produced by the teacher so it can perform useful tasks with less compute, memory, latency, or cost.
Decision for Plain-Language Explanation

Why It Matters

Distillation can make deployment practical, but the student may inherit teacher errors or lose capabilities in ways that require fresh evaluation.
Decision for Why It Matters

Example

A company distills a large language model into a smaller model for a narrow classification task that must run on local hardware.
Decision for Example

Common Misconception

A distilled model is not simply a compressed file or guaranteed miniature copy. It is a separately trained model with its own behavior and limitations.
Decision for Common Misconception

Remember It This Way

The student learns selected behavior from the larger teacher.
Decision for Remember It This Way

Related Terms

* [[foundation-model|Foundation Model]] * [[parameters|Parameters]] * [[compute|Compute]] * [[capability-provenance|Capability Provenance]] * [[robustness|Robustness]]
Decision for Related Terms

Sources

* [Hinton, Vinyals, and Dean, Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531) — primary paper describing knowledge distillation.
Decision for Sources

Learn More

* Record the teacher and training process in [[capability-provenance|Capability Provenance]].
Decision for Learn More
Return to term

Not saved yet.