LV2DMOT: Language and Visual Multimodal Feature Learning for Multiobject Tracking
Ru Hong, Zeyu Cai, Jiming Yang, Feipeng Da · IEEE Sensors Journal · 2025
Multi-object tracking (MOT) aims to associate objects of the same identity across video frames, with robust similarity measurement being crucial for maintaining tracking performance. However, the current inefficient integration of motion and appearance cues often leads to tracking failures in challenging scenarios, such as occlusions and missed detections. In this paper, we introduce LV2DMOT, a tracker that employs a novel paradigm for integrating motion and appearance cues through language and visual multi-modal feature learning, thereby generating more distinctive data association similarities. We propose three key techniques: I) A text matching task between tracking trajectories and candidate detections. This method uses text encoding of detection geometric information combined with a temporal model, Mamba, to extract temporal motion features of trajectories, enabling more accurate motion similarity calculations. II) A multi-modal, multi-level feature fusion model that integrates motion and appearance features via cross modal learning mechanism, resulting in more robust fused similarities. III) A learnable temporal attention model for trajectory appearance feature updates, which effectively aggregates historical visual features to improve the representational ability of trajectory appearance features, employing k-medoids for feature selection. Extensive experiments on the MOT17 and MOT20 datasets demonstrate that our method achieves state-of-the-art tracking performance.