Scalable, Flexible and Active Learning on Distributions
Danica J. Sutherland · 2016
Abstract : A wide range of machine learning problems, including astronomical inferenceabout galaxy clusters, natural image scene classification, parametric statistical inference,and detection of potentially harmful sources of radiation, can be well-modeledas learning a function on (samples from) distributions. This thesis explores problemsin learning such functions via kernel methods, and applies the framework to yieldstate-of-the-art results in several novel settings.One major challenge with this approach is one of computational efficiency whenlearning from large numbers of distributions: the computation of typical methodsscales between quadratically and cubically, and so they are not amenable to largedatasets. As a solution, we investigate approximate embeddings into Euclideanspaces such that inner products in the embedding space approximate kernel valuesbetween the source distributions. We provide a greater understanding of the standardexisting tool for doing so on Euclidean inputs, random Fourier features. We alsopresent a new embedding for a class of information-theoretic distribution distances,and evaluate it and existing embeddings on several real-world applications.The next challenge is that the choice of distance is important for getting goodpractical performance, but how to choose a good distance for a given problem isnot obvious. We study this problem in the setting of two-sample testing, wherewe attempt to distinguish two distributions via the maximum mean divergence, andprovide a new technique for kernel choice in these settings, including the use ofkernels defined by deep learning-type models.In a related problem setting, common to physical observations, autonomoussensing, and electoral polling, we have the following challenge: when observingsamples is expensive, but we can choose where we would like to do so, how do wepick where to observe?