Meta-Learning Adaptation Phase Enhanced Feature Distillation for Accelerated Convergence and Improved Generalization
Xinghan Pan · Preprints.org · 2025
We propose a novel framework that integrates a meta-learning adaptation phase with feature-level knowledge distillation to accelerate convergence and improve the generalization of lightweight neural networks. Our framework aligns intermediate features via a hybrid loss combining mean-squared error and cosine similarity, and refines the student model using a MAML-based meta-learning adaptation phase. Our theoretical analysis, under simplifying assumptions, demonstrates that this dual mechanism reduces generalization error by effectively lowering model complexity and enforcing an information bottleneck. Experimental results on CIFAR-10 confirm that teacher guidance accelerates early training, and our analysis suggests that dynamic scheduling of the distillation weight may further enhance performance.