Distilling Adaptive Instance-Aware Knowledge for Small-footprint Speaker Verification
Meirong Li, Wen Hao Xu · 2023
Knowledge distillation is an effective method for small-footprint speaker verification modeling. However, most existing methods always assume that all input instances are valuable for learning, resulting in limited performance improvement. In this work, we propose an adaptive instance-aware knowledge distillation method that allows small-scale models to selectively learn knowledge from large-scale models. It adaptively transfers not only the embedding-level information of individual instances, but also the correlation between instances. Empirical experiments on the VoxCeleb datasets show that our proposed method can achieve better distillation results, and outperform the baselines by an average of 10.34% in terms of equal error rate (EER).