Sparse Online Variational Bayesian Regression
Kody J. H. Law, Vitaly Zankin · SIAM/ASA Journal on Uncertainty Quantification · 2022
Abstract. This work considers variational Bayesian inference as an inexpensive and scalable alternative to a fully Bayesian approach in the context of sparsity-promoting priors. In particular, the priors considered arise from scale mixtures of normal distributions with a generalized inverse Gaussian mixing distribution. This includes the variational Bayesian LASSO as an inexpensive and scalable alternative to the Bayesian LASSO introduced in T. Park and G. Casella [ J. Amer. Statist. Assoc., 103 (2008), pp. 681–686]. It also includes a family of priors which more strongly promote sparsity. For linear models the method requires only the iterative solution of deterministic least squares problems. Furthermore, for [Formula: see text] unknown covariates the method can be implemented exactly online with a cost of [Formula: see text] in computation and [Formula: see text] in memory per iteration—in other words, the cost per iteration is independent of [Formula: see text], and in principle infinite data can be considered. For large [Formula: see text] an approximation is able to achieve promising results for a cost of [Formula: see text] per iteration in both computation and memory. Strategies for hyperparameter tuning are also considered. The method is implemented for real and simulated data. It is shown that the performance in terms of variable selection and uncertainty quantification of the variational Bayesian LASSO can be comparable to the Bayesian LASSO for problems which are tractable with that method and for a fraction of the cost. The present method comfortably handles [Formula: see text], [Formula: see text] on a laptop in less than 30 minutes, and [Formula: see text], [Formula: see text] overnight.