Sampling Local‐Scale Parameters in High‐Dimensional Regression Models
Anirban Bhattacharya, James E. Johndrow · Wiley StatsRef: Statistics Reference Online · 2021
Abstract A popular prior choice for Bayesian analysis of high‐dimensional regression models is the global–local family of Gaussian scale‐mixture priors. These priors can be seen as alternatives to the more classical spike and slab or “two group” priors that have some advantages in terms of model specification and computation while retaining many of the nice theoretical properties of the spike and slab priors. In this article, we consider computation for the linear model using local–global priors via blocked Gibbs sampling, which is currently the default choice for computation when the number of covariates exceeds the sample size. We focus in particular on the sampling of the local scales within a blocked Gibbs scheme. Our aim is to convey some challenges in implementation for the high‐dimensional setting and the various algorithmic innovations that we have employed to overcome them. These innovations convey the subtlety and delicacy of Bayesian computation for high‐dimensional, sparse models that rarely occur in lower dimensional problems and suggest problems that may be lurking within many implementations of MCMC for these models that can be nontrivial to detect. We focus on the horseshoe prior, which is the most popular and well characterized of the global–local priors, though we expect that many of our conclusions apply to other global–local priors.