Dual-branch Cross-scale Feature Interaction for Temporal Action Detection
Lifang Wu, Xin Chang, Zun Li, Di Cui · Neurocomputing · 2024
Temporal Action Detection (TAD) is aim to predict action boundary and category simultaneously. Most existing RGB-based methods model temporal dependency using pyramid-style features without interaction among different scales, which usually result in inaccurate prediction for long-term actions. The reason is that features at different scales involve information with different granularity , which is suitable for either prediction of action boundary or category. In this paper, we present a novel Dual-branch Cross-scale Feature Interaction (DCFI) method that directly exchanges different scale information from both temporal and spatial perspective for TAD. To be specific, in one branch, a cross-scale temporal transformer module is devised to enable both semantic and temporal communications among different scale features with a merge-to-split mechanism. While the other branch designs a cross-scale spatial mixer module to mine the most salient spatial difference between consecutive and long-term frames via a scale-mixer. Benefiting from these two modules, DCFI achieves comprehensive temporal as well as spatial interaction across all feature scales, and thus accurately predicts the boundaries of different time-span action instances. Extensive experiments on two challenging benchmarks, i.e. ., THUMOS-14 and ActivityNet-1.3, demonstrate that our DCFI achieves new state-of-the-art performance with only RGB.