VLIS: Unimodal Language Models Guide Multimodal Language Generation
Jiwan Chung, Youngjae Yu · 2023
Multimodal language generation, which leverages the synergy of language and vision, is a rapidly expanding field.However, existing vision-language models face challenges in tasks that require complex linguistic understanding.To address this issue, we introduce Visual-Language models as Importance Sampling weights ( VLIS), a novel framework that combines the visual conditioning capability of vision-language models with the language understanding of unimodal text-only language models without further training.It extracts pointwise mutual information of each image and text from a visual-language model and uses the value as an importance sampling weight to adjust the token likelihood from a text-only model.VLIS improves visionlanguage models on diverse tasks, including commonsense understanding (WHOOPS, OK-VQA, and ScienceQA) and complex text generation (Concadia, Image Paragraph Captioning, and ROCStories).Our results suggest that VLIS represents a promising new direction for multimodal language generation. Named EntitiesWho is this?Does he care for his family?