A Unifying Framework of Anytime Sparse Gaussian Process Regression Models with Stochastic Variational Inference for Big Data
Trong Nghia Hoang, Quang Minh Hoang, Bryan Kian Hsiang Low · 2015
This paper presents a novel unifying framework of anytime sparse Gaussian process regression (SGPR) models that can produce good predic-tive performance fast and improve their predic-tive performance over time. Our proposed unify-ing framework reverses the variational inference procedure to theoretically construct a non-trivial, concave functional that is maximized at the pre-dictive distribution of any SGPR model of our choice. As a result, a stochastic natural gradient ascent method can be derived that involves itera-tively following the stochastic natural gradient of the functional to improve its estimate of the pre-dictive distribution of the chosen SGPR model and is guaranteed to achieve asymptotic conver-gence to it. Interestingly, we show that if the pre-dictive distribution of the chosen SGPR model satisfies certain decomposability conditions, then the stochastic natural gradient is an unbiased es-timator of the exact natural gradient and can be computed in constant time (i.e., independent of data size) at each iteration. We empirically eval-uate the trade-off between the predictive perfor-mance vs. time efficiency of the anytime SGPR models on two real-world million-sized datasets. 1.