Bayesian Spatio-Temporal Modelling with Fourier Features
Anthony Tompkins · The Sydney eScholarship Repository (The University of Sydney) · 2018
One of the most powerful machine learning techniques is \emph{Gaussian Processes} (GPs) which incur an $O(N^3)$ complexity in the number of data samples. In regression and classification there exist approximation methods which typically rely on $M$ \emph{inducing points} but still typically incur an $O(NM^2)$ complexity in the data and corresponding inducing points which have reduced expressiveness the larger the dataset becomes. These methods are typically unable to learn if the number of datapoints becomes computationally intractable. It is this limitation of traditional methods that invites us to explore alternative representations of kernels to enable scalable modelling and inference for spatio-temporal phenomena. The key insight we leverage is providing \emph{feature}-space representations of kernels which have computational dependence \emph{independent} of data samples. While such representations exist in various forms, they typically address kernels of infinite support and have not been investigated extensively for modeling periodicity or data supported on bounded intervals. Our approach leverages methods in harmonic analysis to provide an alternative form of representing kernels using Fourier Series which we demonstrate to have superior performance to alternative feature representations. Our methodology further develops \emph{compositional} kernels and show it is straightforward to integrate our Fourier series features with standard kernels. With compositions of kernels we are able to represent nuances in the data that canonical kernels typically cannot represent. This thesis brings the following contributions: 1) A new formulation of representing univariate periodic kernels using Fourier series that allows one to perform scalable inference with a large number of samples; 2) A generalisation of univariate periodic kernels into the multivariate domain which allows tractable higher dimensional inference; 3) An efficient method for the tricky problem of seeding and learning periodic hyperparameters; 4) A generalised framework that allows one to perform compositional kernel learning in a Bayesian framework for spatio-temporal phenomena.