Every route to small, on one size/accuracy chart

Storage bits vs. accuracy for: small dense students, pruned teachers (sparse and structured), quantized versions of each, and the order of operations.

Scatter plot of test accuracy (20-way) against storage (kbit) (log scale), one marker per family, points labeled by config, with a dashed reference line for the teacher. Titled “Every route to small, on one size/accuracy chart”. The numbers are in the table below. — Test accuracy (20-way) against storage (kbit)

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 156 s.

Family	Config	Test accuracy (20-way)	Target-class AP ('bird')	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)	Storage (kbit)
teacher	fp32	0.914	0.989	0.384	0.309	140,052	4481.7
small dense student (KD)	2x8 fp32	0.444	0.741	1.637	1.555	356	11.4
small dense student (KD) + QAT	2x8 8b	0.468	0.773	1.553	1.471	353	4.032
small dense student (KD) + QAT	2x8 4b	0.454	0.761	1.594	1.512	312	2.752
small dense student (KD)	2x16 fp32	0.677	0.860	0.960	0.878	820	26.2
small dense student (KD) + QAT	2x16 8b	0.698	0.872	0.905	0.824	813	8.576
small dense student (KD) + QAT	2x16 4b	0.675	0.857	0.960	0.879	726	5.504
small dense student (KD)	2x32 fp32	0.828	0.953	0.526	0.450	2,132	68.2
small dense student (KD) + QAT	2x32 8b	0.842	0.963	0.475	0.399	2,126	21.1
small dense student (KD) + QAT	2x32 4b	0.836	0.956	0.506	0.429	1,893	12.9
small dense student (KD)	2x64 fp32	0.897	0.986	0.317	0.240	6,292	201.3
small dense student (KD) + QAT	2x64 8b	0.904	0.987	0.303	0.227	6,251	60.0
small dense student (KD) + QAT	2x64 4b	0.901	0.987	0.310	0.234	5,633	35.5
structured-pruned teacher (KD)	3x64 fp32	0.891	0.981	0.322	0.245	10,452	334.5
structured-pruned teacher (KD) + QAT	3x64 4b	0.900	0.984	0.311	0.234	9,287	58.0
structured-pruned teacher (KD)	3x32 fp32	0.790	0.942	0.638	0.556	3,188	102.0
structured-pruned teacher (KD) + QAT	3x32 4b	0.824	0.956	0.540	0.459	2,814	19.1
structured-pruned teacher (KD)	3x16 fp32	0.585	0.801	1.249	1.169	1,092	34.9
structured-pruned teacher (KD) + QAT	3x16 4b	0.629	0.829	1.109	1.025	961	7.296
sparse teacher (KD)	90% fp32	0.913	0.987	0.275	0.198	14,714	610.1
sparse teacher: prune -> QAT 4b	90% 4b	0.914	0.986	0.294	0.217	14,405	359.4
sparse teacher: QAT 4b -> prune	90% 4b	0.913	0.987	0.276	0.199	14,358	358.0
sparse teacher (KD)	97% fp32	0.828	0.955	0.529	0.445	4,966	298.2
sparse teacher: prune -> QAT 4b	97% 4b	0.858	0.968	0.434	0.353	4,817	320.5
sparse teacher: QAT 4b -> prune	97% 4b	0.794	0.946	0.635	0.549	4,868	320.1
sparse teacher (KD)	99% fp32	0.088	0.039	2.943	2.857	2,181	209.1
sparse teacher: prune -> QAT 4b	99% 4b	0.087	0.037	2.937	2.851	2,181	309.3
sparse teacher: QAT 4b -> prune	99% 4b	0.088	0.176	2.922	2.836	2,179	309.3

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.

Birds in this experiment

The table's 27 smaller models hatch 5 kinds of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.

Barn Swallow, healthyFamily: structured-pruned teacher (KD); config: 3x64 fp3213.3× smaller, keeps 97.5% of the teacher's accuracyBest of 4 models
Barn Swallow, starvingFamily: sparse teacher: prune -> QAT 4b; config: 97% 4b13.9× smaller, keeps 93.9% of the teacher's accuracyThe only model of its kind
Ruby-throated Hummingbird, healthyFamily: small dense student (KD) + QAT; config: 2x64 4b126.4× smaller, keeps 98.6% of the teacher's accuracyBest of 4 models
Ruby-throated Hummingbird, starvingFamily: small dense student (KD) + QAT; config: 2x32 8b212.2× smaller, keeps 92.1% of the teacher's accuracyBest of 12 models
Ruby-throated Hummingbird, skeletonFamily: small dense student (KD) + QAT; config: 2x8 4b1628.5× smaller, keeps 49.7% of the teacher's accuracyBest of 6 models

Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, accuracy 0.914. Storage is the table's own column. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.

These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.