SSPT ‐Tr: Self‐Supervised Pre‐Training Transformer Based on Triplet for Temporal Action Detection
Qiongmin Zhang, Zeyuan Deng, Bingyi Ran, Shuqiu Tan, Xin Feng · IEEJ Transactions on Electrical and Electronic Engineering · 2025
Accurate proposal generation is crucial for subsequent classification networks; thus, the temporal action proposal generation (TAPG) methods have a significant influence in the field of Temporal Action Detection. The preparation process of supervised TAPG methods is time‐consuming and resource‐intensive, relying on a large amount of labeled data. Furthermore, due to the relatively small variations in feature sequences at the temporal level in videos, localizing the boundaries of actions is particularly challenging. To address these issues, we first propose a self‐supervised pre‐training method that designs a Random Query Segment Detection pretext task as the learning objective for pre‐training. This enables the training of an action localizer without any annotations. Additionally, when localizing video action segments, the temporal boundaries can be blurred, and the simple feature contrast operation designed during the pre‐training process may not effectively distinguish action boundaries. Therefore, this work introduces an improved method, self‐supervised pre‐training transformer based on triplet (SSPT‐Tr) for feature reconstruction based on triplet to address the aforementioned issue. A negative video segment is added to reconstruct features, and triplet loss is used to further constrain the boundary feature expression capabilities between action and background. This can effectively enhance the feature discrimination between actions and non‐actions. Extensive experiments on the THUMOS14 and ActivityNet‐1.3 datasets demonstrate that the SSPT‐Tr method improves the performance obviously, which not only improves the AR but also shortens the training time of the downstream task. The SSPT‐Tr combined with UNet also outperforms other methods in the field of Temporal Action Detection in terms of mAP at various tIoU thresholds. © 2025 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.