Predicting Cognitive Decline: A Multimodal AI Approach to Dementia Screening from Speech
Lei Chi, Arav Sharma, Ari Gebhardt, Joseph T. Colonel · 2025
Recent progress has been made in detecting early stage dementia entirely through recordings of patient speech. Multimodal speech analysis methods were applied to the Prediction and Recognition Of Cognitive declinE through Spontaneous Speech (PROCESS) Signal Processing Grand Challenge, which requires participants to use audio recordings of clinical interviews to predict patients as healthy control, mild cognitive impairment (MCI), or dementia and regress the patient's MiniMental State Exam (MMSE) scores. The approach implemented in this work combines acoustic features (eGeMAPS and Prosody) with embeddings from Whisper and RoBERTa models, achieving competitive results in both regression (RMSE: 2.7666) and classification (Macro-F1 score: 0.5774) tasks. Additionally, a novel two-tiered classification architecture is utilized to better differentiate between MCI and dementia. Our approach achieved strong results on the test set, ranking seventh on regression and eleventh on classification out of thirty-seven teams, exceeding the baseline performance.