ANALYSIS OF THE ERROR FUNCTION IN CASE OF APPLYING THE AMSGrad OPTIMIZATION ALGORITHM
Serhiy Sveleba, Ivan Katerynchuk, Ivan Kuno, O. Semotyuk, Ya. Shmygelsky, S. Velgosh, A. Kopach, V. Stakhura · Electronics and Information Technologies · 2023
In this paper, the AMSGrad stochastic optimization method was tested using the logistic function that describes the doubling process and the Fourier spectra of the error function. The implementation of the gradient descent optimization algorithm using the AMSGrad method was carried out for a simple two-dimensional function that squares the input data of each measurement and determines the range of acceptable input data from -1.0 to 1.0. The program for minimizing the error function was implemented using Python programming language. The influence of hyperparameters' values beta1 , beta2 , and the learning rate on optimizing the training of this system was analyzed. Also, branching diagrams from these parameters were constructed. We found that the retraining process is accompanied by a change in the rate of the target error function, and the Fourier spectra are characterized by the harmonics' appearance. The instability in the learning process caused by the retraining process is shown to be observed in a small set of input data when the value of beta2 is close to 1 and beta1 = 0.9. Also, we found that a transition accompanies overtraining through the global minimum of the objective function, and multiple passages accompany the transition to chaos through the global minimum. Keywords: optimization methods, error function, AMSGrad, learning rate, branching diagrams.