Knowledge distillation via softmax regression representation learning
Jing Yang, Brais Martínez, Adrian Bulat, Georgios Tzimiropoulos · Queen Mary Research Online (Queen Mary University of London) · 2021
This paper addresses the problem of model compression via knowledge distillation.We advocate for a method that optimizes the output feature of the penultimate layer of the student network and hence is directly related to representation learning.To this end, we firstly propose a direct feature matching approach which focuses on optimizing the student's penultimate layer only.Secondly and more importantly, because feature matching does not take into account the classification problem at hand, we propose a second approach that decouples representation learning and classification and utilizes the teacher's pre-trained classifier to train the student's penultimate layer feature.In particular, for the same input image, we wish the teacher's and student's feature to produce the same output when passed through the teacher's classifier, which is achieved with a simple L 2 loss.Our method is extremely simple to implement and straightforward to train and is shown to consistently outperform previous state-of-the-art methods over a large set of experimental settings including different (a) network architectures, (b) teacher-student capacities, (c) datasets, and (d) domains.