Teacher privileged distillation: How to deal with imperfect teachers?
Mario Martínez-García, Iñaki Inza, Jose Antonio Lozano · Knowledge-Based Systems · 2025
The paradigm of learning using privileged information leverages privileged features present at training time, but not at prediction, as additional training information. The privileged learning process is addressed through a knowledge distillation perspective: information from a teacher learned with regular and privileged features is transferred to a student composed exclusively of regular features. While most approaches assume perfect knowledge for the teacher, it can commit mistakes. Assuming that, we propose a novel privileged distillation framework with a double contribution. Firstly, a designed function to imitate the teacher when it classifies correctly and to differ in cases of misclassification. Secondly, an adaptation of the cross-entropy loss to appropriately penalize the instances where the student outperforms the teacher. Its effectiveness is empirically demonstrated on datasets with imperfect teachers, significantly enhancing the performance of state-of-the-art frameworks. Furthermore, necessary conditions for successful privileged learning are presented, along with a dataset categorization based on the information provided by the privileged features. • Necessary conditions for a successful privileged learning based on information theory. • New categorization of the datasets derived from the information offered by the privileged features. • A novel privileged learning framework, Teacher Privileged Distillation (TPD), to deal with imperfect teachers.