SwapTrack: Enhancing RGB-T Tracking via Learning from Paired and Single-Modal Data
Jianyu Xie, Zhuo Zeng, Zhijie Yang, Junlin Zhou, D.L.Roshni Bai, Duanbing Chen · 2024
RGB-T tracking leverages the complementary information from both RGB and thermal modalities, enhancing tracking robustness and accuracy in challenging visual conditions. Most existing RGB-T trackers primarily rely on paired RGB-T data for training. However, the availability of paired RGB-T images is limited compared to the abundance of single-modal RGB or TIR images. To fully leverage the potential of all available data, we propose SwapTrack, an RGB-T tracker that effectively learns from both paired RGB-T data and single-modal data. The proposed approach incorporates three key designs: shared and separated networks to extract modality-shared and modality-specific patterns, the swapped projection network for modal feature conversion and complementation, and a two-step training scheme to learn from single-modal and multi-modal data effectively. Experimental results demonstrate that the proposed method outperforms state-of-the-art approaches when trained with RGB-T+RGB+TIR datasets. Furthermore, ablation studies reveal that the proposed method demonstrates notable performance enhancements when introducing additional single-modal data. This finding underscores the effectiveness of incorporating single-modal data into the training process.