University of Hildesheim at SemEval-2023 Task 1: Combining Pre-trained Multimodal and Generative Models for Image Disambiguation
Sebastian Diem, Chanjong Im, Thomas Mandl · 2023
Multimodal ambiguity is a challenge for understanding text and images.Large pre-trained models have reached a high level of quality already.This paper presents an implementation for solving an image disambiguation task relying solely on the knowledge captured in multimodal and language models.Within task 1 of SemEval 2023 (Visual Word Sense Disambiguation), this approach managed to achieve an MRR of 0.738 using CLIP-Large and the OPT model for generating text.Applying a generative model to create more text given a phrase with an ambiguous word leads to an improvement in our results.The performance gain from a bigger language model is larger than the performance gain from using the larger CLIP model.