Supervised & unsupervised transfer learning
Julia Elisabeth Vogt · edoc (University of Basel) · 2013
This thesis investigates transfer learning in two areas of data analysis, supervised and unsupervised learning. We study multi-task learning on vectorial data in a supervised setting and multi-view clustering on pairwise distance data in a Bayesian unsupervised approach. The aim in both areas is to transfer knowledge over different related data sets as opposed to learning on single data sets separately. In supervised learning, not only the input vectors but also the corresponding target vectors are observed. The aim is to learn a mapping from the input space to the target space to predict the target values for new samples. In standard classification or regression problems, one data set at a time is considered and the learning problem for every data set is solved separately. In this work, we are looking at the non-standard case of learning by exploiting the information given by multiple related tasks. Multi-task learning is based on the assumption that multiple tasks share some features or structures. One well-known technique solving multi-task problems is the Group-Lasso with 2-norm regularization. The motivation for using the Group-Lasso is to couple the individual tasks via the group-structure of the constraint term. Our main contribution in the supervised learning part consists in deriving a complete analysis of the Group-Lasso for all p-norm regularizations, including results about uniqueness and completeness of solutions and coupling properties of different p-norms. In addition, a highly efficient active set algorithm for all p-norms is presented which is guaranteed to converge and which is able to operate on extremely high-dimensional input spaces. For the first time, this allows a direct comparison and evaluation of all possible Group-Lasso methods for all p-norms in large scale experiments. We show that in a multi-task setting, both, tight coupling norms with p >> 2 and loose coupling norms with p << 2 significantly degrade the prediction performance. Moderate coupling norms seem to be the best compromise between coupling strength and robustness against systematic differences between the tasks. The second area of data analysis we look at is unsupervised learning. In unsupervised learning, the training data consists of input vectors without any corresponding target vectors. Classical problems in unsupervised learning are clustering, density estimation or dimensionality reduction. As in the supervised scenario, we are not only considering single data sets independently of each other, but we want to learn over two or more data sets simultaneously. A problem that arises frequently is that the data is only available as pairwise distances between objects (e.g. pairwise string alignment scores from protein sequences) and a loss-free embedding into a vector space is usually not possible. We propose a Bayesian clustering model that is able to operate on this kind of distance data without explicitly embedding it into a vector space. Our main contribution in the unsupervised learning part is twofold. Firstly, we derive a fully probabilistic clustering method based on pairwise Euclidean distances, that is rotation-, translation-, and scale- invariant and uses the Wishart distribution in the likelihood term. On the algorithmic side, a highly efficient sampling algorithm is presented. Experiments indicate the advantage of encoding the translation invariance into the likelihood, and our clustering algorithm clearly outperforms several hierarchical clustering methods. Secondly, we extend this clustering method to a novel Bayesian multi-view clustering approach based on distance data. We show that the multi-view clustering method reveals shared information between different views of a phenomenon and we obtain an improved clustering compared to clustering on every view separately.