Reference and Document Aware Semantic Evaluation Methods for Korean Language Summarization
Dongyub Lee, Myeong Cheol Shin, Taesun Whang, Seungwoo Cho, Byeongil Ko, Daniel C. Lee, EungGyun Kim, Jaechoon Jo · 2020
Text summarization refers to the process that generates a shorter form of text from the source document preserving salient information.Many existing works for text summarization are generally evaluated by using recall-oriented understudy for gisting evaluation (ROUGE) scores.However, as ROUGE scores are computed based on n-gram overlap, they do not reflect semantic meaning correspondences between generated and reference summaries.Because Korean is an agglutinative language that combines various morphemes into a word that express several meanings, ROUGE is not suitable for Korean summarization.In this paper, we propose evaluation metrics that reflect semantic meanings of a reference summary and the original document, Reference and Document Aware Semantic Score (RDASS).We then propose a method for improving the correlation of the metrics with human judgment.Evaluation results show that the correlation with human judgment is significantly higher for our evaluation metrics than for ROUGE scores.