BAM! Born-Again Multi-Task Networks for Natural Language Understanding
Kevin B. Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, Quoc Viet Le · 2019
It can be challenging to train multi-task neural networks that outperform or even match their single-task counterparts.To help address this, we propose using knowledge distillation where single-task models teach a multi-task model.We enhance this training with teacher annealing, a novel method that gradually transitions the model from distillation to supervised learning, helping the multi-task model surpass its single-task teachers.We evaluate our approach by multi-task fine-tuning BERT on the GLUE benchmark.Our method consistently improves over standard single-task and multi-task training.