Multi-modal Concept Alignment Pre-training for Generative Medical Visual Question Answering
Quan Yan, Junwen Duan, Jianxin Wang · 2024
Medical Visual Question Answering (Med-VQA) seeks to accurately respond to queries regarding medical images, a task particularly challenging for open-ended questions.This study unveils the Multi-modal Concept Alignment Pre-training (MMCAP) approach for generative Med-VQA, leveraging a knowledge graph sourced from medical image-caption datasets and the Unified Medical Language System.MMCAP advances the fusion of visual and textual medical knowledge via a graph attention network and a transformer decoder.Additionally, it incorporates a Type Conditional Prompt in the fine-tuning phase, markedly boosting the accuracy and relevance of answers to open-ended questions.Our tests on benchmark datasets illustrate MMCAP's superiority over existing methods, demonstrating its high efficiency in data-limited settings and effective knowledge-image alignment capability.