Advocating Character Error Rate for Multilingual ASR Evaluation
D K Thennal, Jesin James, Deepa P. Gopinath, Muhammed Ashraf K · 2025
Automatic speech recognition (ASR) systems have traditionally been evaluated using English datasets, with the word error rate (WER) serv ing as the predominant metric.WER's simplic ity and ease of interpretation have contributed to its widespread adoption, particularly for En glish.However, as ASR systems expand to mul tilingual contexts, WER fails in various ways, particularly with morphologically complex lan guages or those without clear word boundaries.Our work documents the limitations of WER as an evaluation metric and advocates for the character error rate (CER) as the primary metric in multilingual ASR evaluation.We show that CER avoids many of the challenges WER faces and exhibits greater consistency across writing systems.We support our proposition by con ducting human evaluations of ASR transcrip tions in three languages-Malayalam, English, and Arabic-which exhibit distinct morpholog ical characteristics.We show that CER corre lates more closely with human judgments than WER, even for English.To facilitate further re search, we release our human evaluation dataset for future benchmarking of ASR metrics.Our findings suggest that CER should be prioritized, or at least supplemented, in multilingual ASR evaluations to account for the varying linguistic characteristics of different languages.