Dual-Feature Attention-Based Contrastive Prototypical Clustering for Multimodal Remote Sensing Data
Shufang Xu, Xinchen Ding, Yiyan Zhang, Zhen Zhang, Hongmin Gao, Bing Zhang · IEEE Transactions on Geoscience and Remote Sensing · 2024
The integrated use of multisource remote sensing (RS) data in Earth observation missions has garnered considerable attention. Hyperspectral images (HSIs) offer extensive spatial and spectral detail, whereas light detection and ranging (LiDAR) data provide elevation information. Therefore, the fusion of HSI and LiDAR data can enhance the accuracy (ACC) of image classification. However, contemporary supervised multimodal deep learning techniques depend heavily on extensive human-annotated training datasets. To address this challenge, we propose a contrastive prototypical clustering network enhanced with a dual-feature attention module. Specifically, two sets of enhanced modal views are constructed from the multimodal RS images for the subsequent contrastive learning. The proposed dual-feature attention module emphasizes channel and spatial attention separately for each modality, integrating both to adjust the feature representation across different channels and positions. By learning the importance weights of each channel and position, this module highlights the hierarchical structure and enhances the discriminative quality of the features. The learned features are utilized through an online clustering mechanism and a self-supervised training strategy that combines contrastive loss and cluster loss to achieve efficient and effective land cover classification. Extensive experiments on three widely used HSI and LiDAR datasets demonstrate that the proposed method outperforms current state-of-the-art approaches. The code for this method is openly available at:https://github.com/RogsDing/DFCPC.