Weight sharing (k-means codebooks) vs. uniform bits

Deep-Compression style: each layer keeps only k distinct values. At the same bits/weight, does a learned, non-uniform codebook beat uniform rounding?

Methods: Weight sharing (k-means codebooks)

Line chart of test accuracy (20-way) against bits / weight, one line per configuration. Titled “teacher”. The numbers are in the table below. — Test accuracy (20-way) against bits / weight: teacher

Note: Both schemes store b bits per weight: k-means has k = 2^b levels per layer, symmetric uniform has 2^b - 1 (1 bit = sign x mean|w|, i.e. 2 levels).

Table

Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 38 s.

Model	Bits / weight	Method	Test accuracy (20-way)	Target-class AP ('bird')	Test NLL	KL(Bayes \|\| model)	Parameters (nonzero)
teacher 3x256	1	uniform (per-tensor)	0.401	0.693	1.950	1.870	140,052
teacher 3x256	1	k-means codebook	0.400	0.713	1.841	1.758	140,052
teacher 3x256	1	k-means + KD + re-snap	0.443	0.731	1.751	1.670	140,052
teacher 3x256	2	uniform (per-tensor)	0.071	0.101	3.021	2.937	3,616
teacher 3x256	2	k-means codebook	0.849	0.979	0.531	0.457	140,052
teacher 3x256	2	k-means + KD + re-snap	0.883	0.984	0.361	0.281	140,052
teacher 3x256	3	uniform (per-tensor)	0.777	0.961	1.225	1.153	60,345
teacher 3x256	3	k-means codebook	0.904	0.983	0.404	0.330	140,052
teacher 3x256	3	k-means + KD + re-snap	0.918	0.988	0.278	0.199	140,052
teacher 3x256	4	uniform (per-tensor)	0.900	0.985	0.466	0.391	102,817
teacher 3x256	4	k-means codebook	0.909	0.987	0.401	0.328	140,052
teacher 3x256	4	k-means + KD + re-snap	0.921	0.988	0.278	0.200	140,052
teacher 3x256	5	uniform (per-tensor)	0.913	0.988	0.390	0.317	122,323
teacher 3x256	5	k-means codebook	0.913	0.988	0.391	0.316	140,052
teacher 3x256	5	k-means + KD + re-snap	0.920	0.990	0.288	0.208	140,052
student 2x32	1	uniform (per-tensor)	0.152	0.537	4.697	4.615	2,132
student 2x32	1	k-means codebook	0.187	0.289	4.420	4.340	2,132
student 2x32	1	k-means + KD + re-snap	0.198	0.274	4.284	4.204	2,132
student 2x32	2	uniform (per-tensor)	0.069	0.043	9.212	9.127	285
student 2x32	2	k-means codebook	0.456	0.714	2.350	2.273	2,132
student 2x32	2	k-means + KD + re-snap	0.464	0.735	2.331	2.254	2,132
student 2x32	3	uniform (per-tensor)	0.485	0.821	2.193	2.114	1,267
student 2x32	3	k-means codebook	0.645	0.846	1.151	1.071	2,132
student 2x32	3	k-means + KD + re-snap	0.668	0.861	1.092	1.013	2,132
student 2x32	4	uniform (per-tensor)	0.723	0.893	0.855	0.776	1,725
student 2x32	4	k-means codebook	0.784	0.948	0.651	0.573	2,132
student 2x32	4	k-means + KD + re-snap	0.784	0.948	0.652	0.575	2,132
student 2x32	5	uniform (per-tensor)	0.813	0.949	0.568	0.494	1,950
student 2x32	5	k-means codebook	0.815	0.951	0.560	0.482	2,132
student 2x32	5	k-means + KD + re-snap	0.827	0.956	0.527	0.449	2,132

Reference points

Teacher (the large model): The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
Bayes-optimal (the best any model can do on this problem): Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.