Evolving Better Initializations For Neural Networks With Pruning
Ryan Zhou, Ting Hu · 2023
Recent work in deep learning has shown that neural networks can be pruned before training to achieve similar or even better results than training the full network. However, existing pruning methods are limited and do not necessarily yield optimal solutions. In this work, we show that perturbing the network by re-initializing the pruned weights and re-pruning can improve performance. We propose to iteratively re-initialize and re-prune using a hill climbing (1 + 1) evolution strategy. We examine the cause of these improvements and show that this method can consistently improve the subnetwork without increasing its size, pointing to a potential new application of evolutionary computing in deep learning.