Calibrating Student Models for Emotion-related Tasks
Mahshid Hosseini, Cornelia Caragea · 2022
Knowledge Distillation (KD) is an effective method to transfer knowledge from one network (a.k.a.teacher) to another (a.k.a.student).In this paper, we study KD on the emotionrelated tasks from a new perspective: calibration.We further explore the impact of the mixup data augmentation technique on the distillation objective and propose to use a simple yet effective mixup method informed by training dynamics for calibrating the student models.Underpinned by the regularization impact of the mixup process by providing better training signals to the student models using training dynamics, our proposed mixup strategy gradually enhances the student model's calibration while effectively improving its performance.We evaluate the calibration of pretrained language models through knowledge distillation over three tasks of emotion detection, sentiment analysis, and empathy detection.By conducting extensive experiments on different datasets, with both in-domain and out-ofdomain test sets, we demonstrate that student models distilled from teacher models trained using our proposed mixup method obtained the lowest Expected Calibration Errors (ECEs) and best performance on both in-domain and outof-domain test sets.