The distillation-loss zoo

Same student, same teacher, same inputs: which form of teacher signal transfers the most? (logits, probabilities, top-k only, hard labels, hidden features, relational structure)

Methods: Distillation loss variants, Feature & relational distillation, Attention transfer

Bar chart of test accuracy (20-way) for each method, with a dashed reference line for the teacher. Titled “The distillation-loss zoo”. The numbers are in the table below. — Test accuracy (20-way) for each method

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 99 s.

Method	Test accuracy (20-way)	Target-class AP ('bird')	Agreement with teacher	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)
labels only (scratch)	0.664 ± 0.009	0.843 ± 0.005	0.674 ± 0.010	1.018 ± 0.027	0.936 ± 0.028	820
hard pseudo-labels	0.671 ± 0.009	0.863 ± 0.003	0.683 ± 0.008	0.993 ± 0.024	0.911 ± 0.024	820
KD T=1	0.671 ± 0.008	0.862 ± 0.002	0.684 ± 0.006	0.991 ± 0.023	0.909 ± 0.023	820
KD T=4	0.640 ± 0.016	0.846 ± 0.009	0.655 ± 0.015	1.819 ± 0.108	1.734 ± 0.108	820
reverse-KL T=4	0.629 ± 0.007	0.818 ± 0.018	0.646 ± 0.007	3.438 ± 0.105	3.349 ± 0.111	820
MSE on logits	0.509 ± 0.005	0.774 ± 0.009	0.524 ± 0.003	2.609 ± 0.017	2.519 ± 0.018	820
top-3 probs only	0.671 ± 0.008	0.863 ± 0.002	0.683 ± 0.007	0.990 ± 0.023	0.908 ± 0.023	820
top-1 prob + uniform rest	0.673 ± 0.006	0.864 ± 0.007	0.686 ± 0.006	0.996 ± 0.020	0.914 ± 0.020	820
KD + FitNets hint	0.672 ± 0.004	0.868 ± 0.011	0.685 ± 0.003	0.989 ± 0.024	0.908 ± 0.025	820
KD + similarity-preserving	0.667 ± 0.006	0.865 ± 0.010	0.678 ± 0.006	0.998 ± 0.027	0.916 ± 0.029	820
hint only (+labels)	0.622 ± 0.010	0.831 ± 0.003	0.636 ± 0.008	1.128 ± 0.038	1.045 ± 0.039	820

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.

Birds in this experiment

The table's 11 smaller models hatch 2 kinds of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.

Ruby-throated Hummingbird, starvingMethod: top-1 prob + uniform rest170.7× smaller, keeps 73.7% of the teacher's accuracyBest of 10 models
Ruby-throated Hummingbird, skeletonMethod: MSE on logits170.7× smaller, keeps 55.6% of the teacher's accuracyThe only model of its kind

Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, accuracy 0.914. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.

These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.