Bag of tricks for a small model

Fixed small architecture. What actually moves its accuracy: regularizers, longer training, weight averaging, over-parameterize-then-collapse, self-distillation, a teacher, or train-big-then-shrink?

Methods: Self-distillation (born-again networks), Over-parameterize, then collapse, Regularizers & training tricks, Train big, then prune to size

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 267 s.

Student	Method	Test accuracy (20-way)	Target-class AP ('bird')	Agreement with teacher	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)
16x16	baseline (labels, CE)	0.664 ± 0.009	0.843 ± 0.005	0.674 ± 0.010	1.018 ± 0.027	0.936 ± 0.028	820
16x16	label smoothing 0.1	0.670 ± 0.008	0.841 ± 0.007	0.679 ± 0.008	1.082 ± 0.021	0.999 ± 0.021	820
16x16	mixup	0.631 ± 0.007	0.838 ± 0.013	0.645 ± 0.008	1.202 ± 0.024	1.118 ± 0.024	820
16x16	EMA of weights	0.648 ± 0.010	0.838 ± 0.006	0.659 ± 0.010	1.060 ± 0.030	0.978 ± 0.032	820
16x16	4x longer training	0.715 ± 0.006	0.880 ± 0.011	0.725 ± 0.005	0.873 ± 0.028	0.792 ± 0.029	820
16x16	over-parameterize, then collapse	0.710 ± 0.004	0.903 ± 0.006	0.718 ± 0.004	0.884 ± 0.009	0.806 ± 0.008	820
16x16	self-distillation (born-again)	0.623 ± 0.007	0.843 ± 0.011	0.637 ± 0.009	1.108 ± 0.010	1.025 ± 0.011	820
16x16	train 2x128, prune to size, self-KD	0.657 ± 0.006	0.848 ± 0.022	0.671 ± 0.008	1.014 ± 0.018	0.931 ± 0.015	820
16x16	KD from teacher (labeled inputs)	0.663 ± 0.009	0.844 ± 0.005	0.674 ± 0.010	1.018 ± 0.028	0.936 ± 0.029	820
16x16	KD from teacher + pool	0.671 ± 0.008	0.862 ± 0.002	0.684 ± 0.006	0.991 ± 0.023	0.909 ± 0.023	820
16x16	KD + pool + online mixup + EMA, 4x long	0.710 ± 0.001	0.880 ± 0.028	0.725 ± 0.002	0.863 ± 0.014	0.780 ± 0.012	820
32x32	baseline (labels, CE)	0.817 ± 0.001	0.952 ± 0.005	0.831 ± 0.003	0.554 ± 0.008	0.476 ± 0.008	2,132
32x32	label smoothing 0.1	0.823 ± 0.004	0.955 ± 0.002	0.832 ± 0.006	0.664 ± 0.013	0.583 ± 0.013	2,132
32x32	mixup	0.799 ± 0.003	0.943 ± 0.007	0.810 ± 0.002	0.754 ± 0.007	0.672 ± 0.008	2,132
32x32	EMA of weights	0.807 ± 0.002	0.948 ± 0.005	0.821 ± 0.004	0.585 ± 0.010	0.506 ± 0.010	2,132
32x32	4x longer training	0.841 ± 0.003	0.965 ± 0.002	0.852 ± 0.000	0.548 ± 0.010	0.471 ± 0.008	2,132
32x32	over-parameterize, then collapse	0.842 ± 0.002	0.964 ± 0.004	0.855 ± 0.001	0.575 ± 0.018	0.503 ± 0.015	2,132
32x32	self-distillation (born-again)	0.801 ± 0.004	0.939 ± 0.006	0.815 ± 0.003	0.595 ± 0.009	0.514 ± 0.007	2,132
32x32	train 2x128, prune to size, self-KD	0.823 ± 0.010	0.949 ± 0.004	0.835 ± 0.010	0.543 ± 0.024	0.466 ± 0.022	2,132
32x32	KD from teacher (labeled inputs)	0.817 ± 0.001	0.952 ± 0.005	0.831 ± 0.003	0.554 ± 0.009	0.475 ± 0.009	2,132
32x32	KD from teacher + pool	0.832 ± 0.005	0.954 ± 0.005	0.845 ± 0.005	0.511 ± 0.013	0.433 ± 0.015	2,132
32x32	KD + pool + online mixup + EMA, 4x long	0.861 ± 0.001	0.972 ± 0.005	0.884 ± 0.002	0.420 ± 0.007	0.341 ± 0.007	2,132

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.

Birds in this experiment

The table's 22 smaller models hatch 1 kind of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.

Ruby-throated Hummingbird, starvingStudent: 32x32; method: KD + pool + online mixup + EMA, 4x long65.6× smaller, keeps 94.2% of the teacher's accuracyBest of 22 models

Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, accuracy 0.914. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.

These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.