Convergence Analysis of Different Categories of Gradient Descent Algorithms
Ruitong Han · 2023
With the rapid development of neural network models, gradient descent as the most important optimization algorithm in neural network models has received extensive attention from academia. At present, neural network technology has been used in various fields, such as computer vision, natural language processing, and speech recognition. In recent years, many kinds of gradient descent algorithms have been proposed to improve existing algorithms, each with higher efficiency and a more stable model structure. This paper will summarize and analyze various gradient descent algorithms and focus on the convergence of various algorithms to find a better method and analyze the optimization ability of that algorithm.