Automatic Generation of Medical Report with Knowledge Graph
Haifeng Zhao, Jie Chen, Lili Huang, Tingting Yang, Wan-Hai Ding, Chuanfu Li · 2021
As an important part of medical diagnosis, medical images are widely used in the diagnosis and treatment of diseases. Radiologists need to write reports for a large number of medical images every day, which usually occupies most of the radiologists’ work time. Automatic medical report generation becomes an urgent problem to be solved. Most of the existing works in the domain of medical report generation have some problems. The first is abnormalities are not well represented and the connection between them is usually ignored. Second, there is no or no good fusion between image and semantic information. We present to establish abnormality graph with prior knowledge to represent abnormalities and their relationships. And we use a multimodal interaction module to fuse image and semantic information. Experimental results on two datasets, a public English medical reports dataset and a Chinese medical reports dataset conducted by ourselves, show that our method outperforms the state-of-art methods.