Patient & consistent teaching

Beyer et al.: treat KD as function matching: query the teacher on fresh augmented inputs every step and train for a long time. Does it keep paying?

Methods: Function matching (patient, consistent teaching)

Line chart of test accuracy (20-way) against training steps (log scale), one line per method, with a dashed reference line for the teacher. Titled “Patient & consistent teaching”. The numbers are in the table below. — Test accuracy (20-way) against training steps

Line chart of agreement with teacher against training steps (log scale), one line per method. Titled “Patient & consistent teaching”. The numbers are in the table below. — Agreement with teacher against training steps

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 302 s.

Training steps	Method	Test accuracy (20-way)	Target-class AP ('bird')	Agreement with teacher	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)
1,500	fixed teacher targets	0.776 ± 0.002	0.922 ± 0.001	0.791 ± 0.002	0.681 ± 0.010	0.601 ± 0.009	2,132
1,500	online mixup (consistent)	0.753 ± 0.008	0.913 ± 0.001	0.771 ± 0.010	0.731 ± 0.021	0.648 ± 0.021	2,132
1,500	online mixup + jitter	0.739 ± 0.011	0.905 ± 0.000	0.760 ± 0.010	0.781 ± 0.025	0.699 ± 0.025	2,132
1,500	labels only (scratch)	0.777 ± 0.001	0.922 ± 0.001	0.791 ± 0.002	0.680 ± 0.009	0.600 ± 0.008	2,132
6,000	fixed teacher targets	0.838 ± 0.001	0.960 ± 0.000	0.850 ± 0.000	0.513 ± 0.004	0.436 ± 0.004	2,132
6,000	online mixup (consistent)	0.838 ± 0.002	0.959 ± 0.003	0.859 ± 0.004	0.484 ± 0.004	0.402 ± 0.004	2,132
6,000	online mixup + jitter	0.832 ± 0.003	0.949 ± 0.009	0.857 ± 0.003	0.498 ± 0.006	0.418 ± 0.007	2,132
6,000	labels only (scratch)	0.836 ± 0.001	0.960 ± 0.000	0.848 ± 0.001	0.518 ± 0.003	0.439 ± 0.003	2,132
24,000	fixed teacher targets	0.838 ± 0.004	0.960 ± 0.001	0.849 ± 0.001	0.675 ± 0.030	0.597 ± 0.026	2,132
24,000	online mixup (consistent)	0.872 ± 0.002	0.974 ± 0.001	0.894 ± 0.002	0.390 ± 0.006	0.310 ± 0.004	2,132
24,000	online mixup + jitter	0.864 ± 0.000	0.971 ± 0.005	0.893 ± 0.001	0.405 ± 0.000	0.324 ± 0.000	2,132
24,000	labels only (scratch)	0.839 ± 0.003	0.965 ± 0.001	0.849 ± 0.001	0.674 ± 0.029	0.601 ± 0.029	2,132

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.

Birds in this experiment

The table's 12 smaller models hatch 2 kinds of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.

Ruby-throated Hummingbird, healthyTraining steps: 24,000; method: online mixup (consistent)65.6× smaller, keeps 95.4% of the teacher's accuracyThe only model of its kind
Ruby-throated Hummingbird, starvingTraining steps: 24,000; method: online mixup + jitter65.6× smaller, keeps 94.6% of the teacher's accuracyBest of 11 models

Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, accuracy 0.914. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.

These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.