DOMAS: Data Oriented Medical Visual Question Answering Using Swin Transformer
Teodora-Alexandra Toader · Studia Universitatis Babeș-Bolyai Informatica · 2023
"The Medical Visual Question Answering problem is a joined Computer Vision and Natural Language Processing task that aims to obtain answers in natural language to a question, posed in natural language as well, regarding an image. Both the image and question are of a medical nature. In this paper we introduce DOMAS, a deep learning model that solves this task on the Med-VQA 2019 dataset. The method is based on dividing the task into smaller classification problems by using a BERT-based question classification and a unique approach that makes use of dataset information for selecting the suited model. For the image classification problems, transfer learning using a pre-trained Swin Transform based architecture is used. DOMAS uses a question classifier and seven image classifiers along with the image classifier selection strategy and achieves 0.616 strict accuracy and 0.654 BLUE score. The results are competitive with other state-of-the-art models, proving that our approach is effective in solving the presented task. 2010 Mathematics Subject Classification. 68T45, 68T50. 1998 CR Categories and Descriptors. I.2.7 [Artificial Intelligence]: Natural Language Processing – Language parsing and understanding; I.2.7 [Artificial Intelligence]: Applications and Expert Systems – Medicine and science. Key words and phrases. Medical Visual Question Answering, Swin Transformer."