Distilling from the true posterior

If the teacher's soft labels were perfect (the exact Bayes posterior), how much better would students get? This bounds what any teacher can give.

Methods: The perfect teacher

Line chart of test accuracy (20-way) against hidden width (log scale), one line per method, with a dashed reference line for the teacher and the Bayes-optimal classifier. Titled “Distilling from the true posterior”. The numbers are in the table below. — Test accuracy (20-way) against hidden width

Line chart of KL(Bayes || model) against hidden width (log scale), one line per method. Titled “Distilling from the true posterior”. The numbers are in the table below. — KL(Bayes || model) against hidden width

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 159 s.

Hidden width	Method	Test accuracy (20-way)	Target-class AP ('bird')	Agreement with teacher	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)
8	labels only (scratch)	0.443 ± 0.015	0.691 ± 0.030	0.450 ± 0.016	1.634 ± 0.027	1.550 ± 0.027	356
8	KD from trained teacher	0.456 ± 0.015	0.740 ± 0.033	0.464 ± 0.018	1.603 ± 0.028	1.519 ± 0.029	356
8	KD from Bayes posterior	0.458 ± 0.014	0.739 ± 0.032	0.465 ± 0.015	1.601 ± 0.027	1.517 ± 0.029	356
8	Bayes-argmax labels on all 28k inputs	0.456 ± 0.013	0.738 ± 0.031	0.464 ± 0.015	1.603 ± 0.026	1.519 ± 0.027	356
16	labels only (scratch)	0.664 ± 0.009	0.843 ± 0.005	0.674 ± 0.010	1.018 ± 0.027	0.936 ± 0.028	820
16	KD from trained teacher	0.671 ± 0.008	0.862 ± 0.002	0.684 ± 0.006	0.991 ± 0.023	0.909 ± 0.023	820
16	KD from Bayes posterior	0.677 ± 0.007	0.863 ± 0.008	0.685 ± 0.004	0.977 ± 0.026	0.894 ± 0.026	820
16	Bayes-argmax labels on all 28k inputs	0.673 ± 0.007	0.856 ± 0.008	0.681 ± 0.007	0.985 ± 0.032	0.903 ± 0.031	820
32	labels only (scratch)	0.817 ± 0.001	0.952 ± 0.005	0.831 ± 0.003	0.554 ± 0.008	0.476 ± 0.008	2,132
32	KD from trained teacher	0.832 ± 0.005	0.954 ± 0.005	0.845 ± 0.005	0.511 ± 0.013	0.433 ± 0.015	2,132
32	KD from Bayes posterior	0.840 ± 0.003	0.959 ± 0.005	0.845 ± 0.003	0.488 ± 0.007	0.409 ± 0.009	2,132
32	Bayes-argmax labels on all 28k inputs	0.838 ± 0.002	0.959 ± 0.006	0.842 ± 0.003	0.495 ± 0.008	0.417 ± 0.010	2,132
64	labels only (scratch)	0.882 ± 0.001	0.980 ± 0.002	0.901 ± 0.002	0.363 ± 0.003	0.286 ± 0.001	6,292
64	KD from trained teacher	0.896 ± 0.001	0.983 ± 0.002	0.916 ± 0.001	0.316 ± 0.003	0.238 ± 0.002	6,292
64	KD from Bayes posterior	0.909 ± 0.003	0.988 ± 0.002	0.910 ± 0.000	0.272 ± 0.003	0.194 ± 0.002	6,292
64	Bayes-argmax labels on all 28k inputs	0.908 ± 0.002	0.988 ± 0.002	0.907 ± 0.001	0.282 ± 0.004	0.206 ± 0.003	6,292
128	labels only (scratch)	0.904 ± 0.002	0.988 ± 0.001	0.932 ± 0.001	0.328 ± 0.008	0.255 ± 0.010	20,756
128	KD from trained teacher	0.916 ± 0.000	0.989 ± 0.000	0.947 ± 0.001	0.267 ± 0.002	0.190 ± 0.001	20,756
128	KD from Bayes posterior	0.938 ± 0.001	0.995 ± 0.000	0.931 ± 0.002	0.188 ± 0.001	0.110 ± 0.001	20,756
128	Bayes-argmax labels on all 28k inputs	0.933 ± 0.000	0.995 ± 0.000	0.929 ± 0.002	0.206 ± 0.002	0.131 ± 0.002	20,756

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.

Birds in this experiment

The table's 20 smaller models hatch 4 kinds of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.

Common Swift, healthyHidden width: 128; method: labels only (scratch)6.7× smaller, keeps 98.9% of the teacher's accuracyBest of 4 models
Ruby-throated Hummingbird, healthyHidden width: 64; method: labels only (scratch)22.2× smaller, keeps 96.5% of the teacher's accuracyBest of 4 models
Ruby-throated Hummingbird, starvingHidden width: 32; method: KD from Bayes posterior65.6× smaller, keeps 91.9% of the teacher's accuracyBest of 8 models
Ruby-throated Hummingbird, skeletonHidden width: 8; method: labels only (scratch)393.4× smaller, keeps 48.5% of the teacher's accuracyBest of 4 models

Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, accuracy 0.914. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.

These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.