KD across the student-size ladder

How much does distillation help at each student size, and how much of that comes from the teacher's soft labels vs. from extra unlabeled inputs?

Methods: Knowledge distillation (Hinton)

Line chart of test accuracy (20-way) against hidden width (log scale), one line per method, with a dashed reference line for the teacher and the Bayes-optimal classifier. Titled “KD across the student-size ladder”. The numbers are in the table below. — Test accuracy (20-way) against hidden width

Line chart of target-class AP ('bird') against hidden width (log scale), one line per method, with a dashed reference line for the teacher and the Bayes-optimal classifier. Titled “KD across the student-size ladder”. The numbers are in the table below. — Target-class AP ('bird') against hidden width

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 157 s.

Hidden width	Method	Test accuracy (20-way)	Target-class AP ('bird')	Agreement with teacher	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)
4	labels only (scratch)	0.315 ± 0.007	0.634 ± 0.011	0.320 ± 0.006	2.016 ± 0.029	1.931 ± 0.029	172
4	KD, labeled inputs	0.315 ± 0.007	0.634 ± 0.009	0.319 ± 0.007	2.016 ± 0.029	1.931 ± 0.029	172
4	KD, + unlabeled pool	0.324 ± 0.012	0.648 ± 0.008	0.329 ± 0.010	1.989 ± 0.040	1.904 ± 0.040	172
8	labels only (scratch)	0.443 ± 0.015	0.691 ± 0.030	0.450 ± 0.016	1.634 ± 0.027	1.550 ± 0.027	356
8	KD, labeled inputs	0.443 ± 0.015	0.690 ± 0.030	0.450 ± 0.016	1.634 ± 0.027	1.550 ± 0.028	356
8	KD, + unlabeled pool	0.456 ± 0.015	0.740 ± 0.033	0.464 ± 0.018	1.603 ± 0.028	1.519 ± 0.029	356
16	labels only (scratch)	0.664 ± 0.009	0.843 ± 0.005	0.674 ± 0.010	1.018 ± 0.027	0.936 ± 0.028	820
16	KD, labeled inputs	0.663 ± 0.009	0.839 ± 0.003	0.673 ± 0.010	1.018 ± 0.027	0.936 ± 0.028	820
16	KD, + unlabeled pool	0.671 ± 0.008	0.862 ± 0.002	0.684 ± 0.006	0.991 ± 0.023	0.909 ± 0.023	820
32	labels only (scratch)	0.817 ± 0.001	0.952 ± 0.005	0.831 ± 0.003	0.554 ± 0.008	0.476 ± 0.008	2,132
32	KD, labeled inputs	0.818 ± 0.000	0.951 ± 0.005	0.833 ± 0.002	0.553 ± 0.008	0.475 ± 0.008	2,132
32	KD, + unlabeled pool	0.832 ± 0.005	0.954 ± 0.005	0.845 ± 0.005	0.511 ± 0.013	0.433 ± 0.015	2,132
64	labels only (scratch)	0.882 ± 0.001	0.980 ± 0.002	0.901 ± 0.002	0.363 ± 0.003	0.286 ± 0.001	6,292
64	KD, labeled inputs	0.882 ± 0.001	0.979 ± 0.002	0.901 ± 0.003	0.361 ± 0.002	0.284 ± 0.000	6,292
64	KD, + unlabeled pool	0.896 ± 0.001	0.983 ± 0.002	0.916 ± 0.001	0.316 ± 0.003	0.238 ± 0.002	6,292
128	labels only (scratch)	0.904 ± 0.002	0.988 ± 0.001	0.932 ± 0.001	0.328 ± 0.008	0.255 ± 0.010	20,756
128	KD, labeled inputs	0.904 ± 0.002	0.988 ± 0.001	0.931 ± 0.003	0.328 ± 0.008	0.255 ± 0.011	20,756
128	KD, + unlabeled pool	0.916 ± 0.000	0.989 ± 0.000	0.947 ± 0.001	0.267 ± 0.002	0.190 ± 0.001	20,756

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.

Birds in this experiment

The table's 18 smaller models hatch 4 kinds of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.

Common Swift, healthyHidden width: 128; method: labels only (scratch)6.7× smaller, keeps 98.9% of the teacher's accuracyBest of 3 models
Ruby-throated Hummingbird, healthyHidden width: 64; method: labels only (scratch)22.2× smaller, keeps 96.5% of the teacher's accuracyBest of 3 models
Ruby-throated Hummingbird, starvingHidden width: 32; method: KD, + unlabeled pool65.6× smaller, keeps 91.0% of the teacher's accuracyBest of 6 models
Ruby-throated Hummingbird, skeletonHidden width: 4; method: labels only (scratch)814.2× smaller, keeps 34.5% of the teacher's accuracyBest of 6 models

Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, accuracy 0.914. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.

These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.