Rethinking Efficient Multilingual Text Summarization Meta-Evaluation
Rilyn Han, Jiawen Chen, Yixin Liu, Arman Cohan · 2024
Evaluating multilingual summarization evaluation metrics, i.e., meta-evaluation, is challenging because of the difficulty of human annotation collection.Therefore, we investigate an efficient multilingual meta-evaluation framework that uses machine translation systems to transform a monolingual meta-evaluation dataset into multilingual versions.To this end, we introduce a statistical test to verify the transformed dataset quality by checking the metaevaluation result consistency on the original dataset and back-translated dataset.With this quality verification method, we transform an existing English summarization meta-evaluation dataset, RoSE, into 30 languages, and conduct a multilingual meta-evaluation of several representative automatic evaluation metrics.In our meta-evaluation, we find that metric performance varies in different languages and neural metrics generally outperform classical textmatching-based metrics in non-English languages.Moreover, we identify a two-stage evaluation method with superior performance, which first translates multilingual texts into English and then performs evaluation.We make the transformed datasets publicly available to facilitate future research.