Temporal Difference Attention for Action Recognition
Hai‐Tao Zhang, Ying Jie Xia · 2024
Temporal difference is a rough motion representation commonly employed in video action recognition frameworks. While its integration can notably enhance the recognition performance of 2D convolutional networks (2D CNNs), it may also encompass substantial regions of action-irrelevant information within the differential map. To this end, this paper proposes a streamlined Temporal Difference Attention Module (TDAM) that combines temporal difference with convolutional spatial attention to extract precise motion representation. Comparative experiments demonstrate that TDAM facilitates ResNet-50 in achieving superior results on Kinetics400 and Activity-Net datasets compared to the recent 2D CNN-based methods (74.87% and 68.71% top-1 accuracy, with only 33.1M parameters, 34.9G FLOPS, 14.6ms latency, and 68.3 FPS). Furthermore, the ablation studies on UCF101 and Activity-Net datasets reveal that the combination of bidirectional difference and convolutional spatial attention is effective and show its good generalization to convolutional architectures.