Why LASSO Seems to Simultaneously Decrease Bias and Variance in Machine Learning
Jochen Merker, Gregor Schuldt · 2021
We show that on an enhancement of the capacity of the function space used in regression, LASSO simultaneously decreases bias and variance of statistical models obtained in machine learning from training data, if the balance between minimization of the mean-squared error and the L1-regularization term is optimal. Further, if minimization of the mean-squared error is dominant, this seems to explain the occurrence of a double descent in the modern interpolation regime of machine learning. Our main method is a decomposition of mean squared error plus complexity into bias, variance and an unavoidable irreducible error inherent to the problem.