Which teacher checkpoint to distill from
Is the most accurate teacher the best teacher? Distill from snapshots along training, plus an over-trained, unregularized teacher.
This figure did not load: the file is missing or the connection dropped. Reload the page to try again; the table below has the same numbers.
This figure did not load: the file is missing or the connection dropped. Reload the page to try again; the table below has the same numbers.
Table
Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 66 s.
| Teacher | Teacher NLL | Teacher ECE | Teacher KL(Bayes || teacher) | Test accuracy (20-way) | Target-class AP ('bird') | Test NLL | KL(Bayes || model) | Teacher accuracy | Parameters (nonzero) |
|---|---|---|---|---|---|---|---|---|---|
| epoch 1 | 0.554 ± 0.000 | 0.051 ± 0.000 | 0.471 ± 0.000 | 0.762 ± 0.002 | 0.921 ± 0.006 | 0.726 ± 0.007 | 0.643 ± 0.006 | 0.818 ± 0.000 | 2,132 |
| epoch 2 | 0.405 ± 0.000 | 0.021 ± 0.000 | 0.324 ± 0.000 | 0.801 ± 0.002 | 0.947 ± 0.003 | 0.613 ± 0.013 | 0.531 ± 0.013 | 0.866 ± 0.000 | 2,132 |
| epoch 4 | 0.341 ± 0.000 | 0.010 ± 0.000 | 0.262 ± 0.000 | 0.814 ± 0.004 | 0.952 ± 0.005 | 0.566 ± 0.015 | 0.488 ± 0.015 | 0.886 ± 0.000 | 2,132 |
| epoch 8 | 0.322 ± 0.000 | 0.031 ± 0.000 | 0.240 ± 0.000 | 0.821 ± 0.003 | 0.957 ± 0.008 | 0.540 ± 0.010 | 0.459 ± 0.011 | 0.900 ± 0.000 | 2,132 |
| epoch 16 | 0.329 ± 0.000 | 0.044 ± 0.000 | 0.255 ± 0.000 | 0.831 ± 0.003 | 0.956 ± 0.003 | 0.514 ± 0.012 | 0.435 ± 0.014 | 0.911 ± 0.000 | 2,132 |
| epoch 32 | 0.367 ± 0.000 | 0.054 ± 0.000 | 0.291 ± 0.000 | 0.832 ± 0.004 | 0.955 ± 0.005 | 0.511 ± 0.013 | 0.433 ± 0.014 | 0.914 ± 0.000 | 2,132 |
| epoch 60 | 0.384 ± 0.000 | 0.055 ± 0.000 | 0.309 ± 0.000 | 0.832 ± 0.005 | 0.954 ± 0.005 | 0.511 ± 0.013 | 0.433 ± 0.015 | 0.914 ± 0.000 | 2,132 |
| 4x longer, no weight decay | 0.551 ± 0.000 | 0.064 ± 0.000 | 0.485 ± 0.000 | 0.831 ± 0.003 | 0.955 ± 0.006 | 0.513 ± 0.011 | 0.434 ± 0.013 | 0.917 ± 0.000 | 2,132 |
| trained on 1k labels | 0.774 ± 0.000 | 0.090 ± 0.000 | 0.698 ± 0.000 | 0.747 ± 0.002 | 0.902 ± 0.011 | 0.808 ± 0.010 | 0.728 ± 0.008 | 0.795 ± 0.000 | 2,132 |
Reference points
- Bayes-optimal (the best any model can do on this problem)
- Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.