Assessing Speech-to-Text Translation Quality: An Overview of Key Metrics
Maria Labied, Abdessamad Belangour, Mouad Banane · 2024
Speech-to-text translation plays a pivotal role in numerous real-world applications, from virtual assistants to live translations. Ensuring high-quality translations requires robust evaluation metrics tailored to different model architectures. This paper presents a detailed review of key metrics for evaluating speech-to-text translation quality. Standard metrics like Character Error Rate, Word Error Rate, and BLEU are introduced, with an explanation of their calculation methods. The paper also explores metrics tailored to cascade models, addressing WER for automatic speech recognition, BLEU, METEOR, and Translation Edit Rate for machine translation, while considering pipeline latency in real-time scenarios. For end-to-end models, metrics such as BLEU, ROUGE, METEOR, and Joint Error Rate are discussed, along with the impact of end-to-end latency. The paper concludes by underscoring the importance of selecting the appropriate evaluation metrics and suggests areas for future research in metric development and model assessment.