Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models
Qinhong Zhou, Zonghan Yang, Peng Li, Yang Liu · 2023
Conventional knowledge distillation (KD) methods require access to the internal information of teachers, e.g., logits.However, such information may not always be accessible for large pre-trained language models (PLMs).In this work, we focus on decision-based KD for PLMs, where only teacher decisions (i.e., top-1 labels) are accessible.Considering the information gap between logits and decisions, we propose a novel method to estimate logits from the decision distributions.Specifically, decision distributions can be both derived as a function of logits theoretically and estimated with test-time data augmentation empirically.By combining the theoretical and empirical estimations of the decision distributions together, the estimation of logits can be successfully reduced to a simple root-finding problem.Extensive experiments show that our method significantly outperforms strong baselines on both natural language understanding and machine reading comprehension datasets.1