A triplet dictionary-driven learning and uncertainty-aware fusion for remote sensing image-text retrieval
Mo Yuan Yang, Luo Chen, Ning Jing · Expert Systems with Applications · 2025
With the development of satellite technology and the proliferation of remote sensing data, developing and designing effective cross-modal image-text retrieval methods has become a popular research topic. The main challenge in this area is to establish effective mapping relationships between visual and textual modalities to form a shared latent space. Existing approaches usually focus on extracting features from each modality independently using pre-trained unimodal models. However, these techniques rarely consider heterogeneous representations of image-text data hindering robust feature extraction and representation to measure high-level semantic similarity, leading to uncertainty struggles between different modalities/objects. To address these limitations, in this paper, we propose a novel Triplet Dictionary-driven learning and Uncertainty-aware Fusion (TDUF) framework, which is end-to-end efficient training of remote sensing cross-model image-text retrieval models. Firstly, a rich feature space is extracted by constructing a structural representation of the Region Adjacency Graph Embedding(RAGE) through local feature description and unsupervised segmentation, and by reducing the data dimensionality in the encoder while preserving the image structural information. Secondly, the Triplet Dictionary-Driven Learning(TDDL) employs a two-layer optimization strategy to construct a sparse metric space, enhancing intra-modal similarity and inter-modal dissimilarity. Finally, Uncertainty-Aware Fusion(UAF) are constructed to quantify the uncertainty of heterogeneous representations using multilevel feature aligners and feature impressions to achieve cross-modal fair comparisons at the object and feature levels. Furthermore, extensive experiments demonstrate that TDUF achieves state-of-the-art performance, outperforming CLIP-based methods by 8.1 % in mean Recall (mR) on remote sensing benchmarks, further enhancement of the ability to obtain geospatial information quickly and accurately.