Knowledge distillation transfers what a large teacher model has learned into a smaller student model by training the student to match the teacher's soft-target probability distributions rather than single labels. Hugging Face's DistilBERT, pretrained with BERT base as teacher, demonstrates the technique in practice: a model that is 40% smaller, retains 97% of language understanding capability, and runs 60% faster.
What Is Knowledge Distillation?
Knowledge distillation is a machine learning technique that transfers what a large, pre-trained "teacher model" has learned into a smaller "student model"CITE:E1. The technique is designed specifically to move learned capability from a bigger network to a more compact one, rather than training the smaller network from scratch on raw labels aloneCITE:E1.
What Are the Core Goal and Training Mechanism Behind Knowledge Distillation?
The primary objective of distillation is training the student network to match the predictions produced by the teacher networkCITE:E2. The teacher's intermediate predictions, known as soft targets, typically serve as the principal training data the student learns fromCITE:E3. In this setup, the student is not simply copying the teacher's final answers but is trained against the richer prediction output the teacher generates during inferenceCITE:E2CITE:E3.
How Is Distillation Loss Measured, and How Do Soft Targets Guide Student Learning?
Distillation loss measures the difference between the probability distribution of the teacher network's soft targets and the probability distribution produced by the student networkCITE:E7. Because each training example can carry multiple soft targets, the student learns the teacher's full probability distribution rather than a single hard labelCITE:E7. This distinction is the mechanical core of the method: the student's training signal is a distribution to match, not a fixed answer to memorizeCITE:E7.
Where Does Knowledge Distillation Come From Academically?
Hinton and collaborators' 2015 paper builds on earlier work by Caruana and colleagues, which had already shown that the knowledge inside an ensemble of models can be compressed into a single model that is far easier to deployCITE:E4. The 2015 paper extends that line of research by applying a different compression technique to achieve the same ensemble-to-single-model transferCITE:E4. This places the teacher-student framework used today within a lineage that runs from ensemble compression to the soft-target training approach described aboveCITE:E4.
How Does DistilBERT Demonstrate Knowledge Distillation in Practice?
DistilBERT, pretrained by Hugging Face using the original BERT base model as its teacher, is a Transformer model built to be smaller and faster than BERTCITE:E6. It was pretrained on the same corpus as BERT in a self-supervised fashion, with BERT base serving as the teacher networkCITE:E6. The measured results of that distillation process are as follows:
| Metric | DistilBERT result |
|---|
| Model size vs. BERT | 40% smallerCITE:E5 |
| Language understanding retained | 97% of BERT's capabilityCITE:E5 |
| Inference speed vs. BERT | 60% fasterCITE:E5 |
These figures come from the same 2019 research describing DistilBERT's compression outcomeCITE:E5.
Taken together, these sources trace a single line from mechanism to product: the loss function that matches probability distributions rather than single labelsCITE:E7 is the same principle the 2015 paper generalized from ensemble compressionCITE:E4, and it is what Hugging Face applied to compress BERT into DistilBERT, a model 40% smaller, 97% as capable, and 60% faster than its teacherCITE:E5CITE:E6.
Author's Take・Nathan
The number worth sitting with here is the 3-percentage-point gap: DistilBERT keeps 97% of BERT's language understanding while cutting size 40% and adding 60% speed, and that gap exists precisely because the training signal is a probability distribution match, not a hard-label copy. That's a meaningfully different optimization target than standard supervised training, which is why distillation can compress an ensemble-derived teacher into a single deployable model instead of just shrinking parameter counts blindly. For anyone evaluating a teacher-student pipeline, the metric to track is whether that retained-capability figure holds up on tasks further from the original pretraining corpus, since the evidence here only establishes it on the benchmark DistilBERT was measured against — the mechanism guarantees distribution matching, not generalized accuracy.