PubMedCLIP: How Much Does CLIP Benefit Visual Question Answering in the Medical Domain?
Sedigheh Eslami, Christoph Meinel, Gerard de Melo · 2023
Contrastive Language-Image Pre-training (CLIP) has shown remarkable success in learning with cross-modal supervision from extensive amounts of image-text pairs collected online.Thus far, the effectiveness of CLIP has been investigated primarily in general-domain multimodal problems.In this work, we evaluate the effectiveness of CLIP for the task of Medical Visual Question Answering (MedVQA).We present PubMedCLIP, a fine-tuned version of CLIP for the medical domain based on PubMed articles.Our experiments conducted on two MedVQA benchmark datasets illustrate that PubMed-CLIP achieves superior results improving the overall accuracy up to 3% in comparison to the state-of-the-art Model-Agnostic Meta-Learning (MAML) networks pre-trained only on visual data.The PubMedCLIP model with different back-ends, the source code for pre-training them and reproducing our MedVQA pipeline is publicly available at https://github.com/sarahESL/PubMedCLIP.