Multimodal Generative Learning Utilizing Jensen-Shannon Divergence
Thomas M. Sutter, Imant Daunhawer, Julia Elisabeth Vogt · Repository for Publications and Research Data (ETH Zurich) · 2019
Learning from different data types is a long standing goal in machine learning research, as multiple information sources co-occur when describing natural phenomena.Existing generative models that try to approximate a multimodal ELBO rely on difficult training schemes to handle the intermodality dependencies, as well as the approximation of the joint representation in case of missing data.In this work, we propose an ELBO for multimodal data which learns the unimodal and joint multimodal posterior approximation functions directly via a dynamic prior.We show that this ELBO is directly derived from a variational inference setting for multiple data types, resulting in a divergence term which is the Jensen-Shannon divergence for multiple distributions.We compare the proposed multimodal JSdivergence (mmJSD) model to state-of-the-art methods and show promising results using our model in unsupervised, generative learning using a multimodal VAE on two different datasets.