Magnitude pruning: the cliff and the regrow
How sparse can the teacher get? Compare one-shot cuts with and without recovery, recovery by labels vs. by distilling from the unpruned self, gradual pruning, and a dense small model with the same weight count.
Methods: Magnitude pruning
This figure did not load: the file is missing or the connection dropped. Reload the page to try again; the table below has the same numbers.
This figure did not load: the file is missing or the connection dropped. Reload the page to try again; the table below has the same numbers.
Table
Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 391 s.
| Fraction of weights removed | Method | Dense width | Test accuracy (20-way) | Target-class AP ('bird') | Test NLL | KL(Bayes || model) | Parameters (nonzero) |
|---|---|---|---|---|---|---|---|
| 0.500 | one-shot, no recovery | 0.896 ± 0.000 | 0.981 ± 0.000 | 0.411 ± 0.000 | 0.331 ± 0.000 | 70,420 | |
| 0.500 | one-shot + label fine-tune | 0.915 ± 0.000 | 0.989 ± 0.000 | 0.396 ± 0.004 | 0.323 ± 0.004 | 70,420 | |
| 0.500 | one-shot + KD from unpruned self | 0.919 ± 0.000 | 0.989 ± 0.000 | 0.312 ± 0.001 | 0.235 ± 0.000 | 70,420 | |
| 0.500 | gradual (5 steps) + KD | 0.917 ± 0.001 | 0.989 ± 0.001 | 0.304 ± 0.001 | 0.224 ± 0.000 | 70,420 | |
| 0.500 | dense small net, same #weights, KD | 248 | 0.920 ± 0.002 | 0.990 ± 0.000 | 0.269 ± 0.002 | 0.193 ± 0.001 | 69,956 |
| 0.750 | one-shot, no recovery | 0.750 ± 0.000 | 0.934 ± 0.000 | 0.771 ± 0.000 | 0.688 ± 0.000 | 35,604 | |
| 0.750 | one-shot + label fine-tune | 0.914 ± 0.000 | 0.988 ± 0.000 | 0.359 ± 0.001 | 0.285 ± 0.002 | 35,604 | |
| 0.750 | one-shot + KD from unpruned self | 0.920 ± 0.001 | 0.989 ± 0.000 | 0.299 ± 0.001 | 0.222 ± 0.001 | 35,604 | |
| 0.750 | gradual (5 steps) + KD | 0.918 ± 0.001 | 0.990 ± 0.000 | 0.274 ± 0.001 | 0.198 ± 0.002 | 35,604 | |
| 0.750 | dense small net, same #weights, KD | 171 | 0.919 ± 0.000 | 0.990 ± 0.001 | 0.265 ± 0.003 | 0.187 ± 0.002 | 35,075 |
| 0.900 | one-shot, no recovery | 0.286 ± 0.000 | 0.636 ± 0.000 | 2.323 ± 0.000 | 2.237 ± 0.000 | 14,714 | |
| 0.900 | one-shot + label fine-tune | 0.902 ± 0.000 | 0.986 ± 0.000 | 0.326 ± 0.001 | 0.250 ± 0.000 | 14,714 | |
| 0.900 | one-shot + KD from unpruned self | 0.912 ± 0.001 | 0.987 ± 0.000 | 0.274 ± 0.001 | 0.198 ± 0.000 | 14,714 | |
| 0.900 | gradual (5 steps) + KD | 0.904 ± 0.001 | 0.987 ± 0.000 | 0.292 ± 0.002 | 0.211 ± 0.002 | 14,714 | |
| 0.900 | dense small net, same #weights, KD | 103 | 0.911 ± 0.001 | 0.987 ± 0.001 | 0.277 ± 0.002 | 0.199 ± 0.002 | 14,131 |
| 0.950 | one-shot, no recovery | 0.121 ± 0.000 | 0.194 ± 0.000 | 2.877 ± 0.000 | 2.792 ± 0.000 | 7,751 | |
| 0.950 | one-shot + label fine-tune | 0.874 ± 0.001 | 0.972 ± 0.000 | 0.384 ± 0.004 | 0.302 ± 0.003 | 7,751 | |
| 0.950 | one-shot + KD from unpruned self | 0.889 ± 0.001 | 0.980 ± 0.000 | 0.343 ± 0.001 | 0.263 ± 0.001 | 7,751 | |
| 0.950 | gradual (5 steps) + KD | 0.858 ± 0.004 | 0.967 ± 0.000 | 0.446 ± 0.008 | 0.367 ± 0.008 | 7,751 | |
| 0.950 | dense small net, same #weights, KD | 69 | 0.898 ± 0.001 | 0.984 ± 0.002 | 0.305 ± 0.004 | 0.229 ± 0.003 | 7,127 |
| 0.980 | one-shot, no recovery | 0.060 ± 0.000 | 0.066 ± 0.000 | 2.991 ± 0.000 | 2.906 ± 0.000 | 3,573 | |
| 0.980 | one-shot + label fine-tune | 0.621 ± 0.005 | 0.825 ± 0.002 | 1.166 ± 0.015 | 1.080 ± 0.015 | 3,573 | |
| 0.980 | one-shot + KD from unpruned self | 0.629 ± 0.000 | 0.832 ± 0.003 | 1.144 ± 0.009 | 1.057 ± 0.008 | 3,573 | |
| 0.980 | gradual (5 steps) + KD | 0.618 ± 0.008 | 0.843 ± 0.005 | 1.209 ± 0.021 | 1.125 ± 0.020 | 3,573 | |
| 0.980 | dense small net, same #weights, KD | 39 | 0.853 ± 0.000 | 0.966 ± 0.004 | 0.447 ± 0.004 | 0.368 ± 0.005 | 2,867 |
| 0.990 | one-shot, no recovery | 0.054 ± 0.000 | 0.038 ± 0.000 | 2.999 ± 0.000 | 2.914 ± 0.000 | 2,181 | |
| 0.990 | one-shot + label fine-tune | 0.080 ± 0.000 | 0.038 ± 0.000 | 2.945 ± 0.000 | 2.859 ± 0.000 | 2,181 | |
| 0.990 | one-shot + KD from unpruned self | 0.088 ± 0.000 | 0.039 ± 0.000 | 2.943 ± 0.000 | 2.857 ± 0.000 | 2,181 | |
| 0.990 | gradual (5 steps) + KD | 0.401 ± 0.009 | 0.642 ± 0.064 | 1.834 ± 0.028 | 1.751 ± 0.028 | 2,181 | |
| 0.990 | dense small net, same #weights, KD | 25 | 0.779 ± 0.004 | 0.925 ± 0.007 | 0.663 ± 0.008 | 0.583 ± 0.006 | 1,495 |
Reference points
- Teacher (the large model)
- The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
- Bayes-optimal (the best any model can do on this problem)
- Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.
Birds in this experiment
The table's 6 smaller models hatch 5 kinds of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.
- Mourning Dove, healthyFraction of weights removed: 0.500; method: dense small net, same #weights, KD; dense width: 2482.0× smaller, scores 100.7% of the teacher's accuracyThe only model of its kind
- Common Swift, healthyFraction of weights removed: 0.750; method: dense small net, same #weights, KD; dense width: 1713.9× smaller, scores 100.5% of the teacher's accuracyThe only model of its kind
- Barn Swallow, healthyFraction of weights removed: 0.900; method: dense small net, same #weights, KD; dense width: 1039.9× smaller, keeps 99.6% of the teacher's accuracyThe only model of its kind
- Ruby-throated Hummingbird, healthyFraction of weights removed: 0.950; method: dense small net, same #weights, KD; dense width: 6919.6× smaller, keeps 98.2% of the teacher's accuracyThe only model of its kind
- Ruby-throated Hummingbird, starvingFraction of weights removed: 0.980; method: dense small net, same #weights, KD; dense width: 3948.8× smaller, keeps 93.3% of the teacher's accuracyBest of 2 models
Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, accuracy 0.914. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.
24 other models have no bird: a magnitude-pruned net also stores where its remaining weights sit, and this table doesn't record that storage.
These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.