RS Glossary
Model Distillation
Model distillation is a training approach in which a smaller model learns to reproduce selected behavior of a larger or more capable teacher model.
- Review state
- Needs review
- Freshness
- Current
- Version
- 2
- Model Distillation
- Knowledge Distillation
Remember it this wayThe student learns selected behavior from the larger teacher.
In plain language
The smaller student model trains on outputs, probabilities, examples, or guidance produced by the teacher so it can perform useful tasks with less compute, memory, latency, or cost.
Why it matters
Distillation can make deployment practical, but the student may inherit teacher errors or lose capabilities in ways that require fresh evaluation.
Example
A company distills a large language model into a smaller model for a narrow classification task that must run on local hardware.
Common misconception
A distilled model is not simply a compressed file or guaranteed miniature copy. It is a separately trained model with its own behavior and limitations.
Related terms
Sources
- Hinton, Vinyals, and Dean, Distilling the Knowledge in a Neural Networkprimary paper describing knowledge distillation.
Learn more
- Record the teacher and training process in [[capability-provenance|Capability Provenance]].
Public history
36 recorded events · 36 public summaries
A potential shared concept was discussed and selected for initial glossary drafting.
The captured candidate received a complete initial draft and entered section-level human review.
Radar reviewed the initial definition draft for human consideration.
Radar reviewed the initial plain-language-explanation draft for human consideration.
Radar reviewed the initial why-it-matters draft for human consideration.
Radar reviewed the initial example draft for human consideration.
Radar reviewed the initial common-misconception draft for human consideration.
Radar reviewed the initial remember-it-this-way draft for human consideration.
Radar reviewed the initial related-terms draft for human consideration.
Radar reviewed the initial sources draft for human consideration.
Radar reviewed the initial learn-more draft for human consideration.
A proposal was opened for section-level human review while preserving the current version.
A human reviewer opened a proposed revision while preserving the current approved or captured version.
definition approved by a human reviewer.
plain-language-explanation approved by a human reviewer.
why-it-matters approved by a human reviewer.
example approved by a human reviewer.
common-misconception approved by a human reviewer.
remember-it-this-way approved by a human reviewer.
related-terms approved by a human reviewer.
sources returned for revision.
learn-more approved by a human reviewer.
Requested wording was prepared without changing approved or existing sections.
Radar reviewed the pending sources section for human consideration.
Radar must rereview the complete model-distillation term after an accepted text change.
Radar must rereview the complete model-distillation term after an accepted text change.
Radar must rereview the complete model-distillation term after an accepted text change.
Radar must rereview the complete model-distillation term after an accepted text change.
Radar must rereview the complete model-distillation term after an accepted text change.
Radar must rereview the complete model-distillation term after an accepted text change.
Radar must rereview the complete model-distillation term after an accepted text change.
Radar must rereview the complete model-distillation term after an accepted text change.
Radar must rereview the complete model-distillation term after an accepted text change.
sources approved by a human reviewer.
sources approved wording became the current section text.
All section gates are approved; content version 2 is now a fully approved term.
Source: vault/Glossary/terms/model-distillation/term.md · sha256:8565367f0ee260b7a86beae63e87057f1c6ff98a587e45c5119fb530c66efd09 · Release: rs-glossary-glossary-team-poc-0.5.5-2026-09-02