Bayesian Model Averaging With Exponentiated Least Squares Loss
Dong Dai, Lei Han, Ting Yang, Tong Zhang · IEEE Transactions on Information Theory · 2018
The model averaging problem is to average multiple models to achieve a prediction accuracy not much worse than that of the best single model in terms of mean-squared error. It is known that if the models are misspecified, model averaging is superior to model selection. Specifically, let$n$be the sample size, then the worst case regret of the former decays at a rate of$O(1/n)$, whereas the worst case regret of the latter decays at a rate of$O(1/\sqrt {n})$. The recently proposed$Q$-aggregation algorithm solves the model averaging problem with the optimal regret of$O(1/n)$both in expectation and in deviation; however, it suffers from two limitations: 1) for continuous dictionary, the proposed greedy algorithm for solving$Q$-aggregation is not applicable and 2) the formulation of$Q$-aggregation appearsad hocwithout clear intuition. This paper examines a different approach to model averaging by considering a Bayes estimator for deviation optimal model averaging by using exponentiated least squares loss. We establish a primal-dual relationship of this estimator and that of$Q$-aggregation and propose new algorithms that satisfactorily resolve the above-mentioned limitations of$Q$-aggregation.