An Overview of Improved Gradient Descent Algorithms for DNN Training within Significant Revolutions of Training Frameworks
Fengcheng Lu · 2021 2nd International Conference on Computing and Data Science (CDS) · 2021
We absolutely need to train parameters for a given Deep Neural Networks (DNN). One of the most common training algorithms is Gradient Descent (GD). This article looks to bring back classical and emerging GD-based algorithms, against poor convergence, for DNN training within transforming training framework-non-distribution/distribution. We also concentrate on asynchronous/synchronous training framework under distributed training. Additionally, we introduce some evolutionary improved methods for deep learning as alternatives to GD, which have proved to be superior to GD in terms of convergence.