Distilling without (much) real data

Which inputs carry the teacher's knowledge best: real labeled, real unlabeled, augmented, pure noise, or inputs dreamed up by inverting the teacher?

Methods: Distilling without real data, DeepInversion (data-free via batch-norm statistics)

Bar chart of test accuracy (20-way) for each source, with a dashed reference line for the teacher. Titled “Distilling without (much) real data”. The numbers are in the table below. — Test accuracy (20-way) for each source

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 107 s.

Source	Test accuracy (20-way)	Target-class AP ('bird')	Agreement with teacher	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)
(labels only, 8k scratch)	0.817 ± 0.001	0.952 ± 0.005	0.831 ± 0.003	0.554 ± 0.008	0.476 ± 0.008	2,132
(labels only, 500 scratch)	0.587 ± 0.001	0.750 ± 0.023	0.595 ± 0.002	2.558 ± 0.040	2.480 ± 0.041	2,132
real: labeled set (8k)	0.818 ± 0.000	0.951 ± 0.005	0.833 ± 0.002	0.553 ± 0.008	0.475 ± 0.008	2,132
real: labeled + unlabeled pool (28k)	0.832 ± 0.005	0.954 ± 0.005	0.845 ± 0.005	0.511 ± 0.013	0.433 ± 0.015	2,132
real: 8k + online mixup	0.813 ± 0.004	0.945 ± 0.004	0.831 ± 0.006	0.564 ± 0.015	0.483 ± 0.015	2,132
real: 8k + online jitter	0.813 ± 0.008	0.944 ± 0.010	0.836 ± 0.008	0.599 ± 0.022	0.517 ± 0.023	2,132
real: 500 only	0.593 ± 0.002	0.749 ± 0.033	0.600 ± 0.003	2.461 ± 0.034	2.383 ± 0.033	2,132
real: 500 + online mixup	0.738 ± 0.004	0.888 ± 0.031	0.754 ± 0.003	0.817 ± 0.024	0.735 ± 0.023	2,132
real: 500 + online mixup+jitter	0.756 ± 0.005	0.903 ± 0.007	0.778 ± 0.005	0.733 ± 0.009	0.651 ± 0.008	2,132
fake: gaussian noise N(0,1)	0.680 ± 0.009	0.853 ± 0.007	0.708 ± 0.010	0.937 ± 0.027	0.854 ± 0.029	2,132
fake: uniform noise [-3,3]	0.679 ± 0.007	0.844 ± 0.010	0.704 ± 0.007	1.053 ± 0.012	0.969 ± 0.012	2,132
fake: teacher inversion (data-free)	0.675 ± 0.003	0.866 ± 0.043	0.699 ± 0.003	1.657 ± 0.019	1.568 ± 0.016	2,132
fake: inversion + 500 real	0.698 ± 0.003	0.859 ± 0.042	0.722 ± 0.006	1.332 ± 0.034	1.246 ± 0.034	2,132

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.

Birds in this experiment

The table's 13 smaller models hatch 1 kind of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.

Ruby-throated Hummingbird, starvingSource: real: labeled + unlabeled pool (28k)65.6× smaller, keeps 91.0% of the teacher's accuracyBest of 13 models

Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, accuracy 0.914. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.

These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.