A stochastic variance reduced gradient using Barzilai-Borwein techniques as second order information
Hardik Tankaria, Nobuo Yamashita · Journal of Industrial and Management Optimization · 2023
In this paper, we consider improving the stochastic variance reduce gradient (SVRG) method by incorporating the curvature information of the objective function. We propose to reduce the variance of stochastic gradients using the computationally efficient Barzilai-Borwein (BB) method by incorporating it into the SVRG. We also incorporate a BB-step size as a variant. We show linear convergence to not only the proposed method but also the other existing SVRG variants that use second-order information. We conduct the numerical experiments on the benchmark datasets and demonstrate that the proposed method with a constant step size outperforms the existing variance reduced methods for some test problems.