Favorable Random Gradients for Optimization of Deep Neural Networks
Yehor Savchenko, Oleksandr Zavalnyi · 2018
We present a technique to improve an optimization of deep neural networks by introducing favorable random gradients during an additional optimization sub-step that affects positively the training process. This technique allows training deep neural networks faster, resulting in smaller training loss. Only the random gradients that do not downgrade the network result on a training mini-batch are selected, and the additional optimization sub-step made by the chosen gradient-based optimizer can be applied. Otherwise, if the random gradients affect poorly, only the standard training step with backpropagation is performed on the given mini-batch.