Robust Dysarthric Speech Recognition with GAN Enhancement and LLM Correction
Yibo He, Kah Phooi Seng, Chee Shen Lim, Li-Minn Ang · Advanced Intelligent Systems · 2025
Dysarthric speech recognition faces significant challenges of acoustic variability and data scarcity, and this study proposes a robust system by integrating generative adversarial network enhancement and large language model correction to address these issues effectively. The system employs three key components, including a multimodal recognition core that combines whisper‐medium encoder with LoRA‐fine‐tuned Llama‐3.1‐8B for end‐to‐end acoustic‐to‐semantic mapping, an improved CycleGAN module that generates synthetic dysarthric speech through Inception‐ResNet fusion blocks, and an intelligent error correction mechanism using N‐best hypothesis reranking with semantic constraints. Experiments on the UA‐Speech dataset show that the complete system achieves a 20.61% word error rate representing a 73.9% relative improvement over traditional end‐to‐end transformer automatic speech recognition. Under very low intelligibility conditions it maintains a 48.69% word error rate demonstrating robust recognition for severe pathological speech. Ablation studies validate each module's effectiveness, providing significant advances for dysarthric patient communication technologies.