A target oriented averaging search trajectory and its application in artificial neural networks
Rojas Jiménez, Ángel Adrián · 2019
The training of artificial neural networks usually involves nonsmooth objective func- tions to be minimized. This optimization problem is currently solved just avoing saddle points and thus reaching a local and fortunately a global minimizer. Well-known opti- mizers like Gradient Descent (GD) or its stochastic version called Stochastic Gradient (SG) depend on the steepest descent direction. Convergence analysis of these methods leave a lot to be desired when is assumed local smoothness and/or (strong) convexity (see, [1][2]). Though, in practice, these iterative methods work well through the back- propagation algorithm [3], we develop a deterministic global optimization method called SALGO-TOAST. Here SALGO denotes the Succesive Abs-Linearization technique of the objective function and the Global Optimization task over that approximation. The latter task is given by our Target Oriented Averaging Search Trajectory (TOAST). Its name try to describe the behavior of the search direction developed in the method. Indeed each direction is defined by an average of the stepest descent direction and oriented by a target value to be reached [4]. The main difference of our algorithm and backpropagation is that the latter does not consider the nonsmoothness of the prediction function. We implement our method to the training of the Rectifier ANNs [5] to solve the learning problem of the Griewank function regression and the digit- image recognition. The results are compared with SG, GD and another deterministic method called Mixed Integer Linear Optimization (MILOP). The last method has a preliminary formulation and is the only one able to reach a global minima [6].