Structured (neuron) pruning: criteria, recovery, budget

Delete whole neurons so the net is genuinely smaller. Which importance criterion picks the right ones, and is a pruned teacher a head start over training the small shape fresh, at the same number of training steps?

Methods: Structured (neuron) pruning, Channel pruning with dependency groups, Prune, then fine-tune (cut and regrow)

Line chart of test accuracy (20-way) against hidden width (log scale), one line per configuration, with a dashed reference line for the teacher. Titled “recovered with KD, full budget (+ fresh init)”. The numbers are in the table below. — Test accuracy (20-way) against hidden width: recovered with KD, full budget (+ fresh init)

Line chart of test accuracy (20-way) against hidden width (log scale), one line per configuration. Titled “no recovery: the cliff”. The numbers are in the table below. — Test accuracy (20-way) against hidden width: no recovery: the cliff

Note: 'short' = 750 steps, 'full' = 3000 steps; pruned and fresh nets use the same optimizer and learning rate, so the only difference is the starting weights.

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 190 s.

Hidden width	Criterion	Recovery	Test accuracy (20-way)	Target-class AP ('bird')	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)
128	l1	none	0.425	0.676	1.990	1.905	37,268
128	l1	KD, short	0.917 ± 0.000	0.989 ± 0.001	0.268 ± 0.000	0.185 ± 0.000	37,268
128	l1	KD, full	0.915 ± 0.000	0.989 ± 0.000	0.315 ± 0.000	0.235 ± 0.000	37,268
128	l1	labels, full	0.909 ± 0.001	0.988 ± 0.000	0.421 ± 0.000	0.346 ± 0.001	37,268
128	act	none	0.405	0.685	2.159	2.081	37,268
128	act	KD, short	0.917 ± 0.001	0.988 ± 0.000	0.263 ± 0.000	0.184 ± 0.000	37,268
128	act	KD, full	0.917 ± 0.001	0.990 ± 0.001	0.301 ± 0.000	0.222 ± 0.000	37,268
128	act	labels, full	0.910 ± 0.001	0.986 ± 0.000	0.408 ± 0.000	0.334 ± 0.001	37,268
128	taylor	none	0.299	0.366	3.864	3.784	37,268
128	taylor	KD, short	0.915 ± 0.000	0.990 ± 0.000	0.272 ± 0.000	0.192 ± 0.000	37,268
128	taylor	KD, full	0.915 ± 0.000	0.990 ± 0.000	0.313 ± 0.001	0.232 ± 0.001	37,268
128	taylor	labels, full	0.906 ± 0.000	0.989 ± 0.000	0.421 ± 0.003	0.341 ± 0.002	37,268
128	random	none	0.309	0.603	2.182	2.093	37,268
128	random	KD, short	0.916 ± 0.000	0.987 ± 0.000	0.262 ± 0.001	0.187 ± 0.000	37,268
128	random	KD, full	0.918 ± 0.002	0.988 ± 0.000	0.308 ± 0.002	0.232 ± 0.002	37,268
128	random	labels, full	0.909 ± 0.000	0.986 ± 0.001	0.404 ± 0.003	0.338 ± 0.002	37,268
128	(fresh init)	KD, short	0.911 ± 0.001	0.989 ± 0.001	0.270 ± 0.002	0.196 ± 0.003	37,268
128	(fresh init)	KD, full	0.918 ± 0.000	0.991 ± 0.001	0.284 ± 0.002	0.210 ± 0.003	37,268
128	(fresh init)	labels, full	0.909 ± 0.003	0.988 ± 0.001	0.382 ± 0.003	0.313 ± 0.002	37,268
64	l1	none	0.169	0.135	2.794	2.711	10,452
64	l1	KD, short	0.893 ± 0.000	0.986 ± 0.000	0.337 ± 0.001	0.255 ± 0.000	10,452
64	l1	KD, full	0.908 ± 0.000	0.987 ± 0.001	0.307 ± 0.000	0.225 ± 0.000	10,452
64	l1	labels, full	0.891 ± 0.001	0.985 ± 0.002	0.434 ± 0.000	0.358 ± 0.000	10,452
64	act	none	0.148	0.478	2.786	2.701	10,452
64	act	KD, short	0.874 ± 0.000	0.977 ± 0.000	0.391 ± 0.001	0.309 ± 0.001	10,452
64	act	KD, full	0.904 ± 0.001	0.987 ± 0.000	0.302 ± 0.001	0.223 ± 0.001	10,452
64	act	labels, full	0.888 ± 0.002	0.980 ± 0.000	0.409 ± 0.000	0.334 ± 0.000	10,452
64	taylor	none	0.152	0.149	3.314	3.233	10,452
64	taylor	KD, short	0.875 ± 0.001	0.971 ± 0.000	0.379 ± 0.001	0.301 ± 0.001	10,452
64	taylor	KD, full	0.903 ± 0.000	0.983 ± 0.000	0.312 ± 0.000	0.235 ± 0.000	10,452
64	taylor	labels, full	0.884 ± 0.000	0.981 ± 0.001	0.428 ± 0.000	0.353 ± 0.000	10,452
64	random	none	0.113	0.228	2.842	2.757	10,452
64	random	KD, short	0.875 ± 0.000	0.975 ± 0.001	0.384 ± 0.001	0.302 ± 0.001	10,452
64	random	KD, full	0.904 ± 0.001	0.986 ± 0.000	0.306 ± 0.003	0.225 ± 0.002	10,452
64	random	labels, full	0.887 ± 0.002	0.982 ± 0.001	0.416 ± 0.008	0.341 ± 0.009	10,452
64	(fresh init)	KD, short	0.869 ± 0.002	0.972 ± 0.003	0.399 ± 0.008	0.319 ± 0.007	10,452
64	(fresh init)	KD, full	0.904 ± 0.001	0.985 ± 0.000	0.299 ± 0.002	0.220 ± 0.003	10,452
64	(fresh init)	labels, full	0.885 ± 0.001	0.979 ± 0.001	0.422 ± 0.008	0.344 ± 0.008	10,452
32	l1	none	0.082	0.197	2.966	2.881	3,188
32	l1	KD, short	0.776 ± 0.003	0.939 ± 0.001	0.687 ± 0.012	0.602 ± 0.012	3,188
32	l1	KD, full	0.860 ± 0.002	0.974 ± 0.003	0.431 ± 0.008	0.346 ± 0.009	3,188
32	l1	labels, full	0.843 ± 0.001	0.971 ± 0.004	0.497 ± 0.002	0.414 ± 0.004	3,188
32	act	none	0.078	0.130	2.969	2.884	3,188
32	act	KD, short	0.732 ± 0.002	0.902 ± 0.002	0.813 ± 0.005	0.731 ± 0.005	3,188
32	act	KD, full	0.848 ± 0.002	0.961 ± 0.001	0.466 ± 0.005	0.387 ± 0.004	3,188
32	act	labels, full	0.827 ± 0.001	0.953 ± 0.002	0.534 ± 0.001	0.454 ± 0.002	3,188
32	taylor	none	0.110	0.082	2.992	2.908	3,188
32	taylor	KD, short	0.721 ± 0.000	0.904 ± 0.002	0.845 ± 0.002	0.763 ± 0.002	3,188
32	taylor	KD, full	0.836 ± 0.002	0.961 ± 0.000	0.504 ± 0.003	0.425 ± 0.003	3,188
32	taylor	labels, full	0.820 ± 0.000	0.951 ± 0.001	0.566 ± 0.001	0.489 ± 0.001	3,188
32	random	none	0.076	0.094	2.987	2.902	3,188
32	random	KD, short	0.731 ± 0.005	0.894 ± 0.002	0.800 ± 0.009	0.716 ± 0.010	3,188
32	random	KD, full	0.848 ± 0.001	0.962 ± 0.000	0.461 ± 0.003	0.379 ± 0.004	3,188
32	random	labels, full	0.826 ± 0.004	0.953 ± 0.002	0.541 ± 0.009	0.460 ± 0.010	3,188
32	(fresh init)	KD, short	0.744 ± 0.002	0.901 ± 0.006	0.765 ± 0.005	0.685 ± 0.007	3,188
32	(fresh init)	KD, full	0.847 ± 0.000	0.968 ± 0.006	0.466 ± 0.000	0.389 ± 0.002	3,188
32	(fresh init)	labels, full	0.825 ± 0.000	0.953 ± 0.004	0.550 ± 0.000	0.474 ± 0.001	3,188
16	l1	none	0.044	0.244	2.996	2.911	1,092
16	l1	KD, short	0.551 ± 0.001	0.747 ± 0.010	1.340 ± 0.003	1.256 ± 0.004	1,092
16	l1	KD, full	0.686 ± 0.006	0.873 ± 0.011	0.941 ± 0.015	0.859 ± 0.014	1,092
16	l1	labels, full	0.680 ± 0.004	0.870 ± 0.017	0.971 ± 0.017	0.890 ± 0.014	1,092
16	act	none	0.059	0.064	3.008	2.923	1,092
16	act	KD, short	0.516 ± 0.005	0.742 ± 0.005	1.411 ± 0.007	1.327 ± 0.006	1,092
16	act	KD, full	0.672 ± 0.002	0.878 ± 0.003	0.983 ± 0.009	0.903 ± 0.009	1,092
16	act	labels, full	0.666 ± 0.000	0.857 ± 0.000	1.013 ± 0.006	0.931 ± 0.006	1,092
16	taylor	none	0.044	0.096	3.033	2.949	1,092
16	taylor	KD, short	0.509 ± 0.000	0.779 ± 0.001	1.442 ± 0.000	1.360 ± 0.000	1,092
16	taylor	KD, full	0.658 ± 0.009	0.846 ± 0.006	1.020 ± 0.026	0.938 ± 0.025	1,092
16	taylor	labels, full	0.642 ± 0.004	0.825 ± 0.001	1.068 ± 0.005	0.983 ± 0.005	1,092
16	random	none	0.055	0.060	2.995	2.911	1,092
16	random	KD, short	0.504 ± 0.003	0.796 ± 0.011	1.453 ± 0.001	1.371 ± 0.001	1,092
16	random	KD, full	0.652 ± 0.007	0.879 ± 0.006	1.025 ± 0.019	0.944 ± 0.020	1,092
16	random	labels, full	0.645 ± 0.002	0.877 ± 0.007	1.044 ± 0.002	0.967 ± 0.001	1,092
16	(fresh init)	KD, short	0.548 ± 0.003	0.793 ± 0.017	1.340 ± 0.015	1.257 ± 0.016	1,092
16	(fresh init)	KD, full	0.701 ± 0.006	0.868 ± 0.012	0.906 ± 0.013	0.825 ± 0.013	1,092
16	(fresh init)	labels, full	0.685 ± 0.001	0.861 ± 0.005	0.955 ± 0.008	0.873 ± 0.007	1,092

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.

Birds in this experiment

The table's 76 smaller models hatch 6 kinds of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.

Common Swift, healthyHidden width: 128; criterion: l1; recovery: KD, short3.7× smaller, scores 100.3% of the teacher's accuracyBest of 15 models
Common Swift, skeletonHidden width: 128; criterion: l1; recovery: none3.7× smaller, keeps 46.5% of the teacher's accuracyBest of 4 models
Barn Swallow, healthyHidden width: 64; criterion: l1; recovery: KD, short13.3× smaller, keeps 97.7% of the teacher's accuracyBest of 15 models
Barn Swallow, skeletonHidden width: 64; criterion: l1; recovery: none13.3× smaller, keeps 18.4% of the teacher's accuracyBest of 4 models
Ruby-throated Hummingbird, starvingHidden width: 32; criterion: l1; recovery: KD, full43.9× smaller, keeps 94.1% of the teacher's accuracyBest of 27 models
Ruby-throated Hummingbird, skeletonHidden width: 16; criterion: l1; recovery: none128.2× smaller, keeps 4.8% of the teacher's accuracyBest of 11 models

Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, accuracy 0.914. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.

These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.