Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting
Po-Yao Huang, Junjie Hu, Xiaojun Chang, Alexander G. Hauptmann · 2020
Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only.However, it is still challenging to associate source-target sentences in the latent space.As people speak different languages biologically share similar visual systems, the potential of achieving better alignment through visual content is promising yet under-explored in unsupervised multimodal MT (MMT).In this paper, we investigate how to utilize visual content for disambiguation and promoting latent space alignment in unsupervised MMT.Our model employs multimodal back-translation and features pseudo visual pivoting in which we learn a shared multilingual visual-semantic embedding space and incorporate visuallypivoted captioning as additional weak supervision.The experimental results on the widely used Multi30K dataset show that the proposed model significantly improves over the state-ofthe-art methods and generalizes well when images are not available at the testing time.