A deepfake video detection method based on multi-modal deep learning method
Yutong Zhang, Xiaoyong Li, Jie Yuan, Yali Gao, Linghui Li · 2021 2nd International Conference on Electronics, Communications and Information Technology (CECIT) · 2021
Recently, most deepfake video classification tasks depend on frame-level features and try to train deep neural networks to characterize fake videos. Although this kind of methods can achieve good results, they also waste the audio information and timing information of the deepfake video dataset. Therefore, to make better use of the audio information, we propose a multi-modal method to detect deepfake videos. The principle of our method is based on the mismatch between audio information and visual information, such as the inconsistency of mouth shape and voice. We calculate the modality dissonance score(MDS score) of videos to classify true/false videos. Extensive experiments reach 84.4% accuracy on the DFDC dataset, which demonstrates the effectiveness of our method.