A systematically wrong teacher
The teacher learned a bias: half its 'bird' training labels said 'sibling'. KD copies the bias faithfully. Can mixing in clean labels undo it?
This figure did not load: the file is missing or the connection dropped. Reload the page to try again; the table below has the same numbers.
This figure did not load: the file is missing or the connection dropped. Reload the page to try again; the table below has the same numbers.
Table
Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 37 s.
| Alpha | Test accuracy (20-way) | Target-class AP ('bird') | Test NLL | KL(Bayes || model) | Parameters (nonzero) |
|---|---|---|---|---|---|
| (biased teacher) | 0.888 | 0.938 | 0.499 | 0.429 | 140,052 |
| 1.0 | 0.794 ± 0.004 | 0.895 ± 0.016 | 0.600 ± 0.010 | 0.521 ± 0.012 | 2,132 |
| 0.9 | 0.797 ± 0.003 | 0.902 ± 0.013 | 0.596 ± 0.009 | 0.517 ± 0.010 | 2,132 |
| 0.5 | 0.811 ± 0.004 | 0.934 ± 0.005 | 0.574 ± 0.010 | 0.494 ± 0.011 | 2,132 |
| 0.25 | 0.815 ± 0.003 | 0.947 ± 0.006 | 0.561 ± 0.008 | 0.481 ± 0.009 | 2,132 |
| 0.0 | 0.817 ± 0.001 | 0.952 ± 0.005 | 0.554 ± 0.008 | 0.476 ± 0.008 | 2,132 |
Reference points
- Teacher (the large model)
- The full-size network trained on the labeled data; the smaller models here are measured against it. Test accuracy (20-way) 0.914, target-class AP ('bird') 0.989, test NLL 0.384, KL(Bayes || model) 0.309, parameters (nonzero) 140,052.
- Bayes-optimal (the best any model can do on this problem)
- Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.
Birds in this experiment
The table's 5 smaller models hatch 1 kind of bird. Each kind is drawn once, for one model: the smallest if the bird is healthy or bones, the one that keeps the most if it's starving.
- Ruby-throated Hummingbird, starvingAlpha: 0.065.6× smaller, keeps 89.4% of the teacher's accuracyBest of 5 models
Measured against this experiment's teacher: 140,052 parameters, 4481.7 kbit, accuracy 0.914. Storage is 32 bits a parameter, the lab's rule for a dense fp32 net, which every model drawn here is. Species follows how many times smaller a model is. A bird stays healthy while its model keeps at least 95% of the reference's score, and turns to bones at 60% or less.
These are the lab's recorded results, so they aren't added to anyone's life list. Birds you hatch yourself, in the Coop or from a Field Guide run, are saved to your life list.