Distilling through label noise

When the labels are noisy, is a teacher trained on those same noisy labels still a better target than the labels themselves?

Line chart of test accuracy (20-way) against label-noise rate, one line per method, with a dashed reference line for the teacher and the Bayes-optimal classifier. Titled “Distilling through label noise”. The numbers are in the table below. — Test accuracy (20-way) against label-noise rate

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 114 s.

Label-noise rate	Method	Test accuracy (20-way)	Target-class AP ('bird')	Agreement with teacher	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)
0	(teacher, noisy labels)	0.914	0.989		0.384	0.309	140,052
0	(teacher, noisy labels, short training)	0.914	0.989		0.265	0.186	140,052
0	labels only (scratch)	0.817 ± 0.001	0.952 ± 0.005	0.831 ± 0.003	0.554 ± 0.008	0.476 ± 0.008	2,132
0	labels + smoothing 0.2	0.816 ± 0.004	0.950 ± 0.002	0.825 ± 0.005	0.824 ± 0.013	0.742 ± 0.013	2,132
0	KD from noisy teacher + pool	0.832 ± 0.005	0.954 ± 0.005	0.845 ± 0.005	0.511 ± 0.013	0.433 ± 0.015	2,132
0	KD from short-trained noisy teacher + pool	0.831 ± 0.002	0.957 ± 0.005	0.845 ± 0.003	0.509 ± 0.011	0.431 ± 0.012	2,132
0.200	(teacher, noisy labels)	0.788	0.904		0.871	0.795	140,052
0.200	(teacher, noisy labels, short training)	0.883	0.981		0.564	0.482	140,052
0.200	labels only (scratch)	0.777 ± 0.002	0.928 ± 0.008	0.788 ± 0.003	0.890 ± 0.013	0.808 ± 0.013	2,132
0.200	labels + smoothing 0.2	0.778 ± 0.004	0.922 ± 0.010	0.788 ± 0.005	1.128 ± 0.013	1.044 ± 0.014	2,132
0.200	KD from noisy teacher + pool	0.798 ± 0.002	0.938 ± 0.010	0.808 ± 0.002	0.778 ± 0.004	0.695 ± 0.005	2,132
0.200	KD from short-trained noisy teacher + pool	0.813 ± 0.006	0.947 ± 0.009	0.824 ± 0.006	0.807 ± 0.009	0.724 ± 0.010	2,132
0.400	(teacher, noisy labels)	0.624	0.759		1.653	1.569	140,052
0.400	(teacher, noisy labels, short training)	0.846	0.957		0.905	0.821	140,052
0.400	labels only (scratch)	0.729 ± 0.002	0.903 ± 0.012	0.736 ± 0.001	1.287 ± 0.004	1.203 ± 0.005	2,132
0.400	labels + smoothing 0.2	0.726 ± 0.005	0.893 ± 0.020	0.734 ± 0.004	1.481 ± 0.004	1.397 ± 0.005	2,132
0.400	KD from noisy teacher + pool	0.753 ± 0.006	0.915 ± 0.015	0.761 ± 0.006	1.135 ± 0.011	1.051 ± 0.012	2,132
0.400	KD from short-trained noisy teacher + pool	0.784 ± 0.006	0.921 ± 0.008	0.792 ± 0.005	1.153 ± 0.007	1.070 ± 0.007	2,132
0.600	(teacher, noisy labels)	0.459	0.483		2.624	2.544	140,052
0.600	(teacher, noisy labels, short training)	0.786	0.893		1.297	1.215	140,052
0.600	labels only (scratch)	0.649 ± 0.012	0.844 ± 0.016	0.658 ± 0.012	1.698 ± 0.020	1.614 ± 0.021	2,132
0.600	labels + smoothing 0.2	0.650 ± 0.017	0.837 ± 0.032	0.658 ± 0.015	1.860 ± 0.028	1.776 ± 0.028	2,132
0.600	KD from noisy teacher + pool	0.673 ± 0.007	0.896 ± 0.016	0.682 ± 0.008	1.540 ± 0.019	1.457 ± 0.019	2,132
0.600	KD from short-trained noisy teacher + pool	0.735 ± 0.002	0.895 ± 0.009	0.744 ± 0.003	1.525 ± 0.006	1.442 ± 0.007	2,132

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.

Birds in this experiment

The table's 16 smaller models hatch 1 kind of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.

Ruby-throated Hummingbird, starvingLabel-noise rate: 0; method: KD from noisy teacher + pool65.6× smaller, keeps 91.0% of the teacher's accuracyBest of 16 models

Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, accuracy 0.914. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.

These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.