Label-smoothed teachers
Label smoothing makes a better model; does it make a better teacher?
This figure did not load: the file is missing or the connection dropped. Reload the page to try again; the table below has the same numbers.
This figure did not load: the file is missing or the connection dropped. Reload the page to try again; the table below has the same numbers.
Table
Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 54 s.
| Teacher label smoothing | Method | Test accuracy (20-way) | Target-class AP ('bird') | Test NLL | KL(Bayes || model) | Teacher accuracy | Parameters (nonzero) |
|---|---|---|---|---|---|---|---|
| 0 | (teacher itself) | 0.914 | 0.989 | 0.384 | 0.309 | 140,052 | |
| 0 | student, KD T=1 | 0.671 ± 0.008 | 0.862 ± 0.002 | 0.991 ± 0.023 | 0.909 ± 0.023 | 0.914 ± 0.000 | 820 |
| 0 | student, KD T=4 | 0.640 ± 0.016 | 0.846 ± 0.009 | 1.819 ± 0.108 | 1.734 ± 0.108 | 0.914 ± 0.000 | 820 |
| 0.100 | (teacher itself) | 0.922 | 0.989 | 0.420 | 0.340 | 140,052 | |
| 0.100 | student, KD T=1 | 0.670 ± 0.006 | 0.868 ± 0.012 | 1.142 ± 0.021 | 1.059 ± 0.021 | 0.922 ± 0.000 | 820 |
| 0.100 | student, KD T=4 | 0.574 ± 0.004 | 0.842 ± 0.019 | 1.531 ± 0.016 | 1.446 ± 0.016 | 0.922 ± 0.000 | 820 |
| 0.200 | (teacher itself) | 0.923 | 0.990 | 0.549 | 0.467 | 140,052 | |
| 0.200 | student, KD T=1 | 0.671 ± 0.007 | 0.864 ± 0.012 | 1.280 ± 0.014 | 1.196 ± 0.014 | 0.923 ± 0.000 | 820 |
| 0.200 | student, KD T=4 | 0.560 ± 0.001 | 0.844 ± 0.009 | 1.724 ± 0.007 | 1.639 ± 0.007 | 0.923 ± 0.000 | 820 |
| 0.400 | (teacher itself) | 0.926 | 0.990 | 0.819 | 0.737 | 140,052 | |
| 0.400 | student, KD T=1 | 0.653 ± 0.002 | 0.863 ± 0.013 | 1.566 ± 0.004 | 1.481 ± 0.004 | 0.926 ± 0.000 | 820 |
| 0.400 | student, KD T=4 | 0.555 ± 0.013 | 0.825 ± 0.017 | 1.955 ± 0.009 | 1.871 ± 0.009 | 0.926 ± 0.000 | 820 |
Reference points
- Bayes-optimal (the best any model can do on this problem)
- Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.