Deleting whole layers: plain vs. residual

Remove k of 8 hidden layers from a deep teacher. Residual nets are said to behave like ensembles of shallow paths. Does that make depth removable? And is initializing a shallow student from the teacher's layers worth it?

Methods: Layer dropping, ShortGPT (block influence)

Line chart of test accuracy (20-way) against layers removed (of 8), one line per configuration. Titled “Deleting whole layers: plain vs. residual”. The numbers are in the table below. — Test accuracy (20-way) against layers removed (of 8)

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 138 s.

Kind	Layers removed (of 8)	Recovery	Init	Test accuracy (20-way)	Target-class AP ('bird')	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)
plain	0	none		0.910	0.985	0.538	0.471	119,828
plain	1	none		0.062	0.126	6.579	6.480	103,316
plain	1	KD		0.916 ± 0.002	0.986 ± 0.002	0.339 ± 0.001	0.268 ± 0.002	103,316
plain	2	none		0.066	0.147	5.151	5.065	86,804
plain	2	KD		0.916 ± 0.001	0.987 ± 0.001	0.300 ± 0.000	0.228 ± 0.001	86,804
plain	4	none		0.053	0.041	6.490	6.399	53,780
plain	4	KD		0.917 ± 0.001	0.988 ± 0.000	0.280 ± 0.001	0.205 ± 0.000	53,780
plain	6	none		0.057	0.041	4.219	4.130	20,756
plain	6	KD		0.914 ± 0.001	0.989 ± 0.000	0.266 ± 0.001	0.189 ± 0.001	20,756
residual	0	none		0.911	0.985	0.394	0.335	119,828
residual	1	none		0.867	0.978	0.497	0.419	103,316
residual	1	KD		0.917 ± 0.001	0.988 ± 0.000	0.297 ± 0.002	0.231 ± 0.002	103,316
residual	2	none		0.804	0.934	0.639	0.560	86,804
residual	2	KD		0.918 ± 0.001	0.987 ± 0.001	0.294 ± 0.003	0.227 ± 0.003	86,804
residual	4	none		0.532	0.679	1.488	1.405	53,780
residual	4	KD		0.917 ± 0.001	0.988 ± 0.000	0.274 ± 0.001	0.205 ± 0.001	53,780
residual	6	none		0.258	0.114	2.594	2.510	20,756
residual	6	KD		0.912 ± 0.000	0.989 ± 0.000	0.260 ± 0.001	0.187 ± 0.000	20,756
			every-other teacher layer	0.916 ± 0.001	0.987 ± 0.000	0.297 ± 0.000	0.231 ± 0.002	53,780
			random	0.919 ± 0.002	0.989 ± 0.000	0.267 ± 0.002	0.193 ± 0.001	53,780

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.910, target-class AP ('bird') 0.985, test NLL 0.538, KL(Bayes || model) 0.471, parameters (nonzero) 119,828.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.

Birds in this experiment

The table's 18 smaller models hatch 8 kinds of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.

Rock Pigeon, healthyKind: plain; layers removed (of 8): 2; recovery: KD1.3× smaller, scores 100.7% of the teacher's accuracyBest of 5 models
Rock Pigeon, starvingKind: residual; layers removed (of 8): 2; recovery: none1.3× smaller, keeps 88.4% of the teacher's accuracyThe only model of its kind
Dodo, skeletonKind: plain; layers removed (of 8): 1; recovery: none1.1× smaller, keeps 6.8% of the teacher's accuracyThe only model of its kind
Rock Pigeon, skeletonKind: plain; layers removed (of 8): 2; recovery: none1.3× smaller, keeps 7.3% of the teacher's accuracyThe only model of its kind
Mourning Dove, healthyKind: plain; layers removed (of 8): 4; recovery: KD2.2× smaller, scores 100.8% of the teacher's accuracyBest of 4 models
Mourning Dove, skeletonKind: plain; layers removed (of 8): 4; recovery: none2.2× smaller, keeps 5.9% of the teacher's accuracyBest of 2 models
Common Swift, healthyKind: plain; layers removed (of 8): 6; recovery: KD5.7× smaller, scores 100.5% of the teacher's accuracyBest of 2 models
Common Swift, skeletonKind: plain; layers removed (of 8): 6; recovery: none5.7× smaller, keeps 6.2% of the teacher's accuracyBest of 2 models

Measured against this experiment's teacher, the plain 8x128 net: 119,828 parameters, 3834.5 kbit, accuracy 0.910. The residual rows are measured against it too, as the reference points are. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.

These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.