Radiology Report Generation for Rare Diseases via Few-shot Transformer
Xing Jia, Yun Xiong, Jiawei Zhang, Yao Zhang, Blackley Suzanne, Yangyong Zhu, Chunlei Tang · 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · 2021
Reliable automatic radiology report generation is highly desired to reduce the labor-intensive and error-prone workload for healthcare workers. While some multi-modal learning models have been proposed to study on this task, few of them paid attention to the radiology report generation for rare diseases, except for RareGen which solved this problem by enhancing the semantic representations of rare diseases. However, there still exist several open problems to be addressed. The first lies in the low proportion of disease regions in an image, making the visual information redundant or irrelevant to rare diseases to be encoded. The second lies in that correlations modeled in the encoding stage may not be effectively decoded in the decoding stage due to the multi-modal representation. To address these two issues, we propose a few-shot Transformer radiology report generation model, namely TransGen, for rare diseases. It integrates the advantages of Transformer with two key modules assembled. Specifically, in the encoding stage, a Semantic-aware Visual Learning (SVL) module is introduced to capture the regions of rare diseases. Following that, in the decoding stage, a Memory Augmented Semantic Enhancement (MASE) module is proposed to enhance intermediate representations. It could make full use of the semantic information contained in the historical-generated sentences to benefit report generation involving rare diseases. Extensive experiments have been conducted on two public datasets of IU X-Ray and MIMIC-CXR to demonstrate the effectiveness of our proposed model.