Distilling the Knowledge in a Neural Network¶

1. Basic Information¶

• Authors: Geoffrey Hinton, Oriol Vinyals, Jeff Dean
• Paper status: NIPS 2014 Deep Learning Workshop
• Link: https://arxiv.org/abs/1503.02531

2. Summary¶

2.1. Key Idea¶

• Can use unlabeled data to transfer knowledge, but using the same training data seems to work well in practice.
• Use softmax with temperature, values from 1-10 seem to work well, depending on the problem.
• The MNIST networks learn to recognize digits without ever having seen base, solely based on the "errors" that the teacher network makes. (Bias needs to be adjusted)

2.2. Contributions¶

• Training on soft targets with less data performs much better than training on hard targets with same amount of data.

3. Explanation¶

• PR12에서 정영재님이 멋지게 발표해주신 내용이 있으니 이걸 참고하도록 합시다.

4. TensorFlow Implementation¶

무엇을 만들어야 하는가? *