MorphNet Impacts on Neural Network Optimizers
M. Aldiki Febriantono, Ridho Herasmara · 2021 3rd International Conference on Cybernetics and Intelligent System (ICORIS) · 2021
Field of artificial intelligence have reported remarkable progress since the 2012. With this progress, computational requirement increases, along with power requirement and carbon footprint. MorphNet proposes solution to increase efficiency of neural network by proposing improved network design from existing network design. We study the impact of MorphNet on optimizer by comparing training process using six optimizers of Adam, AdaMax, AdaGrad, AdaDelta, RMSProp, and SGD. We constructed models according to proposed model from previous MorphNet research, and performed training using the optimizers. As the result, training time using selected optimizers for 20 epochs, on average reduced by 19%. Validation loss on average was increased by 19.65%. Validation accuracy on average was reduced by 1.20 percentage points. On training time, SGD optimizer was the most improved with reduction in training time of 29%. On validation loss, AdaGrad and AdaDelta was the most affected with increase of 33.6% and 33.7% respectively. On validation accuracy, the AdaGrad and AdaDelta was also the most affected with reduction of 1.97 and 4.97 percentage points respectively. However, we found that AdaMax performs the best on the smallest network size in terms of training time and accuracy.