End‐To‐End Multiple Object Detection and Tracking With Spatio‐Temporal Transformers
Qi Lei, Xiangyu Song, Shijie Sun, Huansheng Song, Lichen Liu, Zhaoyang Zhang · IET Computer Vision · 2025
ABSTRACT Optimising both trajectory position information and identity information is a key challenge in multiple object tracking. Mainstream approaches ensure ID consistency by combining detection data with various additional information. However, many methods overlook the inherent spatio‐temporal correlation of trajectory position information. We argue that additional modules are redundant, and that forecasting trajectories directly without the need for interframe association by utilising motion constraints is adequate. In this study, we introduce a novel end‐to‐end network called the spatio‐temporal multiple object tracking with transformer (STMOTR), which employs motion constraints to establish binary matching within the reconstructed deformable‐DETR network, heuristically learning object trajectories from the Video Swin backbone. This subtly constrained matching rule not only keeps the detection ID consistency but also significantly reduces the potential for tracking ID switch. We evaluated STMOTR on the UA‐DETRAC and our proposed tunnel multiple object tracking dataset (T‐MOT), achieving state‐of‐the‐art performance with 39.8% PR‐MOTA on the UA‐DETRAC and 79.6% MOTA on the T‐MOT. The source code is also available at https://github.com/Jade‐Ray/STMOTR .