AI Empathetic Chatbot with Real-Time Emotion Detection Using Multimodal Fusion and BO-CNN Optimization
Shine P Xavier, Saju P. John · 2024
This paper presents a novel AI-driven empathetic chatbot system that leverages multimodal fusion to decode and respond to user emotions across speech, text, and visual cues in real-time. Building upon advanced Speech Emotion Recognition (SER) and Face Emotion Recognition, and integrating BERT-based Natural Language Understanding (NLU), the system is designed to enhance human-like interaction and emotional responsiveness in conversational AI. A comprehensive review of literature highlights limitations in existing approaches, such as single-modal reliance and dataset constraints, which our proposed model addresses by fusing multiple data sources using the EmbraceNet framework. This fusion of facial and speech features is further optimized with Botox Optimization-based Convolutional Neural Networks (BO-CNN), enhancing accuracy in facial emotion recognition. Extensive testing validates the model's superior performance across precision, sensitivity, and specificity metrics, demonstrating its potential for application in various human-interactive systems.