MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
Qinxin Wang, Hao Xuan Tan, Sheng Shen, Michael W. Mahoney, Zhewei Yao · 2020
Phrase localization is a task that studies the mapping from textual phrases to regions of an image.Given difficulties in annotating phraseto-object datasets at scale, we develop a Multimodal Alignment Framework (MAF) to leverage more widely-available caption-image datasets, which can then be used as a form of weak supervision.We first present algorithms to model phrase-object relevance by leveraging fine-grained visual representations and visually-aware language representations.By adopting a contrastive objective, our method uses information in caption-image pairs to boost the performance in weakly-supervised scenarios.Experiments conducted on the widely-adopted Flickr30k dataset show a significant improvement over existing weaklysupervised methods.With the help of the visually-aware language representations, we can also improve the previous best unsupervised result by 5.56%.We conduct ablation studies to show that both our novel model and our weakly-supervised strategies significantly contribute to our strong results. 1