SARNet: Self-attention Assisted Ranking Network for Temporal Action Proposal Generation
Jiahao Yu, Hong Jiang · 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC) · 2021
Temporal action detection is a fundamental yet challenging video understanding task. The calculation of confidence score for each generated action proposals remains the bottleneck of this task. Given that the continuity of videos is beneficial for self-supervised learning, in this paper we propose Self-attention Assisted Ranking Network (SARNet), which uses a self-attention mechanism to assist the ranking and retrieval of generated proposals. Our method incorporates a discriminative and a generative constraint to train the self-attention weight. Extensive experiments on THUMOS14 demonstrate that our method achieves a considerable improvement of average recall with a small number of proposals, and brings the mAP up to 30% at tIoU threshold 0.7 for the first time.