Polyshrink: An Adaptive Variable Selection Procedure That Is Competitive with Bayes Experts
Dean P. Foster, Robert A. Stine · 2005
We propose an adaptive shrinkage estimator for use in regression problems chara- terized by many predictors, such as wavelet estimation. Adaptive estimators perform well over a variety of circumstances, such as regression models in which few, some or many coecients are zero. Our estimator, PolyShrink, adaptively varies the amount of shrinkage to suit the esti- mation task. Whereas hard thresholding using the risk inflation criterion is optimal for models with few predictors, PolyShrink obtains a broader competitive optimality vis-a-vis the best Bayes expert. A Bayes expert is the predictive distribution implied by a prior distribution for the unknown coecients. We derive non-asymptotic upper and lower bounds for the expected log-loss, or divergence, of Bayes experts whose prior is unimodal and symmetric about zero. Our bounds hold for any sample size and are pointwise in the sense that they hold for any values of the unknown parameters. These bounds allow us to show that PolyShrink produces a fitted model whose divergence lies within a constant factor of the divergence obtained by the best Bayes expert. In a simulation of four frequently considered wavelet estimation problems, PolyShrink obtains smaller mean squared error than hard thresholding, which is not adaptive, and several other adaptive estimators.