Carving a specialist out of the generalist

Prune the teacher's neurons for one class. Does a task-aware importance score (bird-vs-rest) keep a better bird detector than an all-class score, and does collapsing the head help recovery?

Methods: Task-aware pruning

Line chart of target-class AP ('bird') against hidden width (log scale), one line per method, with a dashed reference line for the teacher. Titled “Carving a specialist out of the generalist”. The numbers are in the table below. — Target-class AP ('bird') against hidden width

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 90 s.

Hidden width	Method	Test accuracy (20-way)	Target-class AP ('bird')	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)	Overlap
128	all-class importance, no recovery	0.299	0.366	3.864	3.784	37,268
128	bird importance, no recovery	0.165	0.758	8.475	8.393	37,268
128	all-class importance, 20-way KD	0.915 ± 0.000	0.990 ± 0.000	0.313 ± 0.001	0.232 ± 0.001	37,268
128	all-class importance, collapsed head + grouped KD		0.990 ± 0.000			35,720
128	bird importance, collapsed head + grouped KD		0.986 ± 0.000			35,720
128	fresh 3xW specialist, grouped KD		0.989 ± 0.002			35,720
64	all-class importance, no recovery	0.152	0.149	3.314	3.233	10,452
64	bird importance, no recovery	0.071	0.533	5.998	5.918	10,452
64	all-class importance, 20-way KD	0.903 ± 0.000	0.983 ± 0.000	0.312 ± 0.000	0.235 ± 0.000	10,452
64	all-class importance, collapsed head + grouped KD		0.988 ± 0.001			9,672
64	bird importance, collapsed head + grouped KD		0.986 ± 0.001			9,672
64	fresh 3xW specialist, grouped KD		0.987 ± 0.000			9,672
32	all-class importance, no recovery	0.110	0.082	2.992	2.908	3,188
32	bird importance, no recovery	0.071	0.500	3.308	3.226	3,188
32	all-class importance, 20-way KD	0.836 ± 0.002	0.961 ± 0.000	0.504 ± 0.003	0.425 ± 0.003	3,188
32	all-class importance, collapsed head + grouped KD		0.975 ± 0.001			2,792
32	bird importance, collapsed head + grouped KD		0.968 ± 0.002			2,792
32	fresh 3xW specialist, grouped KD		0.967 ± 0.000			2,792
16	all-class importance, no recovery	0.044	0.096	3.033	2.949	1,092
16	bird importance, no recovery	0.078	0.510	2.937	2.853	1,092
16	all-class importance, 20-way KD	0.658 ± 0.009	0.846 ± 0.006	1.020 ± 0.026	0.938 ± 0.025	1,092
16	all-class importance, collapsed head + grouped KD		0.895 ± 0.001			888
16	bird importance, collapsed head + grouped KD		0.892 ± 0.001			888
16	fresh 3xW specialist, grouped KD		0.908 ± 0.000			888
8	all-class importance, no recovery	0.068	0.098	3.014	2.929	428
8	bird importance, no recovery	0.077	0.531	2.971	2.887	428
8	all-class importance, 20-way KD	0.463 ± 0.009	0.704 ± 0.010	1.613 ± 0.023	1.527 ± 0.022	428
8	all-class importance, collapsed head + grouped KD		0.641 ± 0.007			320
8	bird importance, collapsed head + grouped KD		0.729 ± 0.005			320
8	fresh 3xW specialist, grouped KD		0.749 ± 0.018			320
4	all-class importance, no recovery	0.054	0.081	2.999	2.914	192
4	bird importance, no recovery	0.063	0.440	2.988	2.904	192
4	all-class importance, 20-way KD	0.319 ± 0.001	0.531 ± 0.008	2.030 ± 0.007	1.943 ± 0.007	192
4	all-class importance, collapsed head + grouped KD		0.488 ± 0.000			132
4	bird importance, collapsed head + grouped KD		0.485 ± 0.017			132
4	fresh 3xW specialist, grouped KD		0.380 ± 0.098			132
128							0.555
64							0.328
32							0.156
16							0.125
8							0.125
4							0

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.

Birds in this experiment

The table's 36 smaller models hatch 8 kinds of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.

Common Swift, healthyHidden width: 128; method: all-class importance, collapsed head + grouped KD3.9× smaller, scores 100.2% of the teacher's target-class APBest of 4 models
Common Swift, starvingHidden width: 128; method: bird importance, no recovery3.7× smaller, keeps 76.7% of the teacher's target-class APThe only model of its kind
Common Swift, skeletonHidden width: 128; method: all-class importance, no recovery3.7× smaller, keeps 37.0% of the teacher's target-class APThe only model of its kind
Barn Swallow, healthyHidden width: 64; method: all-class importance, 20-way KD13.3× smaller, keeps 99.5% of the teacher's target-class APThe only model of its kind
Barn Swallow, skeletonHidden width: 64; method: all-class importance, no recovery13.3× smaller, keeps 15.0% of the teacher's target-class APBest of 2 models
Ruby-throated Hummingbird, healthyHidden width: 32; method: all-class importance, collapsed head + grouped KD50.1× smaller, keeps 98.7% of the teacher's target-class APBest of 7 models
Ruby-throated Hummingbird, starvingHidden width: 16; method: fresh 3xW specialist, grouped KD157.7× smaller, keeps 91.8% of the teacher's target-class APBest of 8 models
Ruby-throated Hummingbird, skeletonHidden width: 4; method: all-class importance, collapsed head + grouped KD1061.0× smaller, keeps 49.4% of the teacher's target-class APBest of 12 models

Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, target-class AP 0.989. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.

These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.