Ridge Regression Based on Gradient Descent Method with Memory Dependent Derivative
Fanming Huang, Dan Li, Jiachen Xu, Yutao Wu, Yidan Yedda Xing, Zan Yang · 2020
Ridge regression is a supervised biased estimation regression method in machine learning. In fact, its principle is similar to that of ordinary least squares(OLS) estimation, except that OLS is improved by giving up the unbiasedness of OLS plus a penalty term, which solves the problem of multicollinearity (approximate linear correlation of independent variables), between data. Ridge regression algorithms are the same in optimizing the loss function of linear regression, and usually use gradient descent or stochastic gradient descent. However, when dealing with complex nonlinear problems, these methods can easily fall into the trap of local minima, and global consciousness and convergence are not strong. In order to retain the gradient information well and enhance the global convergence, the gradient descent method of memory-dependent derivative is introduced into the optimized loss function part of the ridge regression algorithm.