Two-Dimensional Attentive Fusion for Multi-Modal Learning of Neuroimaging and Genomics Data
Md Abdur Rahaman, Yash Garg, Armin Iraji, Zening Fu, Jiayu Chen, Vince D. Calhoun · 2022
Human exposure to reality is multi-modal, and the brain processes it through multi-sensory stimulation. As such, using multi-source intelligence can potentially improve results motivated by human learning. The key challenge in multi-modal learning is to integrate the modalities through a sensible fusion. We propose mBAM - a novel fusion technique inspired by the bottleneck attention module (BAM) to leverage the knowledge from diverse data modes. We combine this module with a deep multi-modal framework for classifying mental disorders. The joint architecture extracts relevant features from diverse inputs - from brain imagery to genomic variables to classify schizophrenia. The model's prediction accuracy is 95.6% (P < 0.0001), outperforming state-of-the-art unimodal and multi-modal models for the task. Moreover, the scheme provides inherent interpretability that helps identify concepts significant for the neural network's decision and explains the underlying factors of the diseases.