Violence Video Detection Based on Multi-modal Fusion and Dual Channel Contrastive Learning
Huan Li, Zhijian Bai, Yue Li, Tao Qin · 2023
There are more and more violence videos propagated on the Internet, it is important to detect them to provide support for online social network management. In the traditional detection methods, the utilization of different modal data is insufficient. In this paper, we propose a violence detection method based on multimodal fusion and dual channel contrastive learning. Firstly, an adaptive feature fusion method based on the attention mechanism is proposed. The problem of original information missing in the fused features is effectively solved by adaptively weighting the original modal features and the attention fusion features. And then the similarity of the same data representations is improved by contrastive learning between different feature fusion methods, in turn, the local problems caused by different fusion methods are solved. Experimental results based on public datasets show that the proposed method can learn the audio and video features and then generate high quality multimodal fusion features, the detection accuracy is improved by 1.3% to 4.8% compared with popular violence detection methods.