Fast Deep Learning Training through Intelligently Freezing Layers
Xueli Xiao, Thosini Bamunu Mudiyanselage, Chunyan Ji, Jie Hu, Yi Feng Pan · 2019
As the complexity of deep learning models grows, the difficulty and time of training them also increases. Depending on the task, the complexity of the model, and the hardware resources available, the amount of training time could take hours, weeks or even months. To decrease the training time, we propose a method to intelligently freeze layers during the training process. Our method involves designing a formula to calculate normalized gradient differences for all layers with weights in the model, and then use the calculated values to decide how many layers should be frozen. We implemented our method on top of stochastic gradient descent, and performed experiments on standard image classification dataset CIFAR-10. Results show that our method can accelerate training on VGG nets, ResNets, and DenseNets while having similar test accuracy.