Medical Visual Textual Entailment for Numerical Understanding of Vision-and-Language Models
Hitomi Yanaka, Yuta Nakamura, Yuki Chida, Tomoya Kurosawa · 2023
Assessing the capacity of numerical understanding of vision-and-language models over images and texts is crucial for real vision-andlanguage applications, such as systems for automated medical image analysis.We provide a visual reasoning dataset focusing on numerical understanding in the medical domain.The experiments using our dataset show that current vision-and-language models fail to perform numerical inference in the medical domain.However, the data augmentation with only a small amount of our dataset improves the model performance, while maintaining the performance in the general domain. *Equal Contribution. 1 https://radiopaedia.org/cases/haemorrhagic-intracranial-metastases-from-breast-cancer