Multimodal Cognitive Learning for Media Forgery Detection: A Comprehensive Framework Combining Random Forest and Deep Ensemble Architectures (Xception, ResNeXt) across Image, Video, and Audio Modalities
A Abirami, S. Bhuvaneswari, K Krithika, I Nithyasree, B Prashithaa Abhirami · 2023
Deepfake content has become more prevalent in the age of quickly evolving technology, which has significantly undermined the reliability and integrity of digital media. An integrated multimodal deepfake detection system is presented in this study as a response to the ubiquitous threat posed by altered photos, videos, and audio recordings. The image deepfake detection module examines visual data for telltale signs of manipulation using Convolutional Neural Networks (CNNs), Xception, and ResNeXT. This module successfully distinguishes between real and fake photos by carefully examining pixel-level attributes and contextual data. With the use of spatiotemporal CNNs (Xception & ResNeXT), it parses video frames to find minute discrepancies, making it possible to accurately identify deepfake films. This multi-modal system is finished with the addition of deepfake audio detection. This module excels in differentiating between authentic and faked audio recordings using Mel spectrograms and Convolutional Neural Networks, adding to a thorough protection against audio deepfakes. Additionally, a unifying framework has been provided that effectively unifies these three detection modules, boosting the system’s effectiveness and performance as a whole. The solution has been thoroughly assessed using measures like accuracy, F1 score, ROC curve, and AUC, and the model structures for in-depth comprehension. This multi-modal deepfake detection technology acts as a crucial precaution in a time when false information is widely disseminated, enabling consumers to distinguish fact from fiction across numerous media types. This study highlights the importance of integrated solution in maintaining the legitimacy of digital content in today’s information-driven world while also showcasing its technological capability.