Language-focused Deepfake Detection Using Phonemes, Mouth Movements, and Video Features
Jonas Krause, Andrei de Souza Inácio, Heitor Silvério Lopes · 2023
The potential implications of Artificial Intelligence (AI) and Deep Learning (DL) algorithms in generating highly realistic deepfake videos have raised concerns regarding the reliability of our human senses. In response to this challenge, we propose a deepfake detection system based on phonemes, the transcribed text, associated mouth movements, and video-extracted features. As a proof-of-concept, we develop a deepfake detection system specifically designed for the Portuguese language, employing three presidential candidates from the 2022 Brazilian elections. Additionally, we introduce a unique dataset comprising real and fake videos involving these three individuals and deliberately blending their identities. The extracted features consolidate relevant attributes, which we utilized to train multiple classification algorithms. Notably, our computational models demonstrate satisfactory performance when authenticating or detecting fake videos containing at least one of the trained phonemes from the Portuguese language. Hence, we conclude that deepfake detection is feasible, primarily due to the absence of natural expressions, particularly in non-English language deepfake videos. Furthermore, developing individual-guided deepfake detection systems may facilitate the authentication of videos featuring celebrities or politicians during future online events.