Medical Visual Question‐Answering Model Based on Knowledge Enhancement and Multi‐Modal Fusion
Dianyuan Zhang, Chuanming Yu, Lu An · Proceedings of the Association for Information Science and Technology · 2024
ABSTRACT This paper aims to utilize a knowledge graph for importing external knowledge. It combines multi‐modal fusion mechanisms and confidence detection mechanisms to explore the correlation between clinical problems and medical images, enhancing their effectiveness in medical visual question‐answering tasks. The proposed medical visual question answering model comprises a text knowledge enhancement layer, an image embedding layer, a multimodal fusion layer, a confidence detection layer, and a prediction layer. The experimental results demonstrate that the medical vision question‐answering model, based on knowledge enhancement and multi‐modal fusion, achieves an optimal accuracy of 59.3% and 16.2% in open‐domain question‐answering tasks on the VQA‐RAD and PathVQA datasets, respectively, thus validating the effectiveness of the proposed model.