Experimental Evaluation of Metaheuristic Optimization of Gradients as an Alternative to Backpropagation
Oleksandr Zavalnyi, Gang Zhao, Yehor Savchenko, Wenlei Xiao · 2018
In this paper, we continue our research on the previously proposed “favorable gradients”, the gradients selected by random search that are beneficial for training of DNNs. The contribution of this study can be stated as follows: we experimentally analyze two methods of calculating gradients - backpropagation and favorable gradients optimization - and we show that the gradients can be optimized by metaheuristic optimization techniques such as evolution strategies and further used for training DNNs with a selected optimizer. Thus, the proposed method can be viewed as an alternative to backpropagation, since it is theoretically possible to train DNNs end-to-end by using only favorable gradients. However, in practice this method turns out to be considerably slower than backpropagation, therefore a combination of these two techniques has been examined.