Knowledge Distillation for Model-Agnostic Meta-Learning
Min Zhang, Wang Donglin, Sibo Gai · Frontiers in artificial intelligence and applications · 2020
Recently, model-agnostic meta-learning (MAML) and its variants have drawn much attention in few-shot learning. In this paper, we investigate how to improve the performance of a portable MAML network so that it can be used in handheld devices, such as small robots, mobile phones, and laptops. We propose a novel approach named portable model-agnostic meta-learning (P-MAML), where valuable knowledge is distilled from a teacher MAML network to a portable student MAML. Moreover, data augmentation and ResNet architecture are employed in the teacher MAML network so as to avoid overfitting and enhance efficiency. To the best of our knowledge, this is the first work to consider a portable meta-learning model through knowledge distillation (KD) to learn a good initialization. Extensive experimental results on three real datasets show that our P-MAML algorithm greatly enhances the accuracy through KD from the teacher network. As shown, P-MAML with KD improves the performance of one-shot learning as high as 10% in comparison to that without KD.