Capacity gap & teacher assistants
Does a bigger teacher always make a better student? When the gap is huge, does an intermediate 'teacher assistant' help?
Methods: Teacher assistants
This figure did not load: the file is missing or the connection dropped. Reload the page to try again; the table below has the same numbers.
This figure did not load: the file is missing or the connection dropped. Reload the page to try again; the table below has the same numbers.
This figure did not load: the file is missing or the connection dropped. Reload the page to try again; the table below has the same numbers.
Table
Mean over the seeds; ± is the standard deviation where more than one seed ran. The experiment took 88 s.
| Teacher width | Student | Path | Test accuracy (20-way) | Target-class AP ('bird') | Test NLL | KL(Bayes || model) | Teacher accuracy | Parameters (nonzero) |
|---|---|---|---|---|---|---|---|---|
| 16 | student 2x8 | 0.461 ± 0.015 | 0.708 ± 0.024 | 1.591 ± 0.039 | 1.506 ± 0.039 | 0.679 ± 0.000 | 356 | |
| 16 | student 2x32 | 0.674 ± 0.002 | 0.862 ± 0.005 | 0.985 ± 0.002 | 0.903 ± 0.003 | 0.679 ± 0.000 | 2,132 | |
| 32 | student 2x8 | 0.470 ± 0.017 | 0.730 ± 0.026 | 1.564 ± 0.016 | 1.480 ± 0.016 | 0.816 ± 0.000 | 356 | |
| 32 | student 2x32 | 0.796 ± 0.005 | 0.925 ± 0.010 | 0.626 ± 0.006 | 0.546 ± 0.006 | 0.816 ± 0.000 | 2,132 | |
| 64 | student 2x8 | 0.460 ± 0.019 | 0.738 ± 0.032 | 1.601 ± 0.030 | 1.517 ± 0.031 | 0.882 ± 0.000 | 356 | |
| 64 | student 2x32 | 0.827 ± 0.003 | 0.954 ± 0.006 | 0.525 ± 0.008 | 0.446 ± 0.009 | 0.882 ± 0.000 | 2,132 | |
| 128 | student 2x8 | 0.455 ± 0.017 | 0.738 ± 0.031 | 1.602 ± 0.031 | 1.518 ± 0.032 | 0.904 ± 0.000 | 356 | |
| 128 | student 2x32 | 0.831 ± 0.003 | 0.956 ± 0.007 | 0.515 ± 0.008 | 0.435 ± 0.008 | 0.904 ± 0.000 | 2,132 | |
| 256 | student 2x8 | 0.456 ± 0.015 | 0.740 ± 0.033 | 1.603 ± 0.028 | 1.519 ± 0.029 | 0.914 ± 0.000 | 356 | |
| 256 | student 2x32 | 0.832 ± 0.005 | 0.954 ± 0.005 | 0.511 ± 0.013 | 0.433 ± 0.015 | 0.914 ± 0.000 | 2,132 | |
| 512 | student 2x8 | 0.456 ± 0.015 | 0.739 ± 0.031 | 1.608 ± 0.031 | 1.525 ± 0.033 | 0.923 ± 0.000 | 356 | |
| 512 | student 2x32 | 0.835 ± 0.001 | 0.955 ± 0.005 | 0.505 ± 0.008 | 0.427 ± 0.009 | 0.923 ± 0.000 | 2,132 | |
| 512 | chain | 512 -> 8 (direct) | 0.456 ± 0.015 | 0.739 ± 0.031 | 1.608 ± 0.031 | 1.525 ± 0.033 | 356 | |
| 512 | chain | 512 -> 128 -> 8 | 0.458 ± 0.015 | 0.738 ± 0.031 | 1.601 ± 0.031 | 1.517 ± 0.032 | 356 | |
| 512 | chain | 512 -> 32 -> 8 | 0.468 ± 0.014 | 0.728 ± 0.013 | 1.573 ± 0.026 | 1.489 ± 0.026 | 356 |
Reference points
- Bayes-optimal (the best any model can do on this problem)
- Predictions from the exact class probabilities, which the lab can compute because it generated the data. Test accuracy (20-way) 0.973, target-class AP ('bird') 0.999, test NLL 0.079.