Students don't match their teachers

Even a student big enough to copy the teacher exactly often doesn't. How does agreement with the teacher grow with capacity and training length?

Line chart of test accuracy (20-way) against training steps (log scale), one line per student, with a dashed reference line for the teacher. Titled “Students don't match their teachers”. The numbers are in the table below. — Test accuracy (20-way) against training steps

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 213 s.

Student	Training steps	Test accuracy (20-way)	Target-class AP ('bird')	Agreement with teacher	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)
16x16	750	0.542 ± 0.002	0.776 ± 0.024	0.554 ± 0.002	1.346 ± 0.016	1.263 ± 0.019	820
16x16	3,000	0.677 ± 0.000	0.860 ± 0.000	0.688 ± 0.001	0.979 ± 0.019	0.898 ± 0.019	820
16x16	12,000	0.738 ± 0.006	0.893 ± 0.003	0.749 ± 0.005	0.797 ± 0.013	0.716 ± 0.012	820
64x64	750	0.839 ± 0.002	0.960 ± 0.000	0.856 ± 0.002	0.494 ± 0.003	0.413 ± 0.004	6,292
64x64	3,000	0.895 ± 0.001	0.983 ± 0.003	0.916 ± 0.002	0.319 ± 0.001	0.240 ± 0.000	6,292
64x64	12,000	0.905 ± 0.002	0.985 ± 0.001	0.934 ± 0.002	0.333 ± 0.003	0.255 ± 0.003	6,292
256x256x256	750	0.921 ± 0.002	0.990 ± 0.001	0.954 ± 0.002	0.246 ± 0.005	0.169 ± 0.006	140,052
256x256x256	3,000	0.922 ± 0.001	0.989 ± 0.000	0.965 ± 0.004	0.291 ± 0.002	0.214 ± 0.002	140,052
256x256x256	12,000	0.922 ± 0.001	0.990 ± 0.000	0.963 ± 0.002	0.301 ± 0.000	0.225 ± 0.001	140,052

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.

Birds in this experiment

The table's 6 smaller models hatch 3 kinds of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.

Ruby-throated Hummingbird, healthyStudent: 64x64; training steps: 3,00022.2× smaller, keeps 98.0% of the teacher's accuracyBest of 2 models
Ruby-throated Hummingbird, starvingStudent: 64x64; training steps: 75022.2× smaller, keeps 91.8% of the teacher's accuracyBest of 3 models
Ruby-throated Hummingbird, skeletonStudent: 16x16; training steps: 750170.7× smaller, keeps 59.3% of the teacher's accuracyThe only model of its kind

Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, accuracy 0.914. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.

These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.