An Adaptive Quasi-Hyperbolic Momentum Method Based on AdaGrad+ Strategy
Hongxu Wei, Xu Zhang, Zhi Fang · 2022 International Conference on Image Processing, Computer Vision and Machine Learning (ICICML) · 2022
The adaptive momentum method is a research hotspot in the field of optimization in recent years. At present, most adaptive momentum methods use AdaGrad-type strategy combined with Heavy-Ball or Nesterov accelerated gradient momentum(NAG). However, the AdaGrad-type strategy does not perform well in constrained optimization, and the Heavy-Ball and NAG momentum cannot reduce the gradient variance and bias at the same time. To solve the above problems, researchers propose AdaGrad+ and quasi-hyperbolic momentum (QHM). In this paper, a QHM momentum method based on AdaGrad+ will be proposed, combining the AdaGrad+ strategy of adjusting the step size adaptively with the QHM method, giving it the advantages of both AdaGrad+ and QHM The iterative formula of the method is given and briefly analyzed in this paper. The feasibility of the method is verified by general convex optimization experiments. In addition, we also conduct deep neural network training experiments to confirm that the method has good performance in practical applications.