Model Selection for Gaussian Process Regression by Approximation Set Coding
Fischer, Benjamin · Repository for Publications and Research Data (ETH Zurich) · 2016
Gaussian processes are powerful, yet analytically tractable models for supervised learning.As a generalization of the multivariate Gaussian distribution, a Gaussian process is characterized by a mean function and a covariance function.The problem of model selection is to determine a mean and covariance function with the aim of adapting the Gaussian process to given data points -a difficult balancing act between data fit and model complexity.The functions to be compared are in essence arbitrary, since they do not just differ in their parametrization but in their fundamental structure.In domains such as systems biology it is often not clear which function structure to choose, for instance to decide between a squared exponential and a rational quadratic covariance function.Based on the theory of approximation set coding (ASC), a framework for model selection is developed, which is general enough to do hyperparameter optimization for any model that has a prior on its parameters.The framework is then applied to Gaussian process regression.Experiments on synthetic and real-world data are presented to compare approximation set coding to the classic model selection criteria of maximum evidence (also known as marginal likelihood) and leave-one-out crossvalidation.Although approximation set coding shows promise to become a competitive model selection criterion, it currently seems not to perform better than the classic criteria in our experiments.Maximum evidence has the best performance in general, while approximation set coding occasionally surpasses leave-one-out cross-validation.Further work is needed on systematic ways to compare model selection criteria, or even combine them in ensembles.My advisers Yatao Bian and Stefan Bauer deserve a lot of gratitude for their continuous inspiration.Our discussions made me look at problems from different points of view, which was crucial to overcome unclear phases of the project.This thesis would not be the same without Nico S. Gorbach.He took all the time in the world to explain things, with his help being of key importance during the whole process.Lastly, I would like acknowledge how my supervisor Professor