M2SGD: Learning to Learn Important Weights
Nicholas I-Hsien Kuo, Mehrtash Tafazzoli Harandi, Nicolas Fourrier, Christian Walder, Gabriela Ferraro, Hanna Suominen · 2020
Meta-learning concerns rapid knowledge acquisition. One popular approach cast optimisation as a learning problem and it has been shown that learnt neural optimisers updated base learners more quickly than their handcrafted counterparts. In this paper, we learn an optimisation rule that sparsely updates the learner parameters and removes redundant weights. We present Masked Meta-SGD (M2SGD), a neural optimiser which is not only capable of updating learners quickly, but also capable of removing 83.71% weights for ResNet20s.We release our codes at https://github.com/Nic5472K/CLVISION2020_CVPR_M2SGD.