On the Evaluation Metrics for Paraphrase Generation
Lingfeng Shen, Lemao Liu, Haiyun Jiang, Shuming Shi · 2022
In this paper we revisit automatic metrics for paraphrase evaluation and obtain two findings that disobey conventional wisdom:(1) Reference-free metrics achieve better performance than their reference-based counterparts.(2) Most commonly used metrics do not align well with human annotation.Underlying reasons behind the above findings are explored through additional experiments and in-depth analyses.Based on the experiments and analyses, we propose ParaScore, a new evaluation metric for paraphrase generation.It possesses the merits of referencebased and reference-free metrics and explicitly models lexical divergence.Based on our analysis and improvements, our proposed reference-based outperforms than referencefree metrics.Experimental results demonstrate that ParaScore significantly outperforms existing metrics.Our codes and toolkit are released in https://github.com/ shadowkiller33/ParaScore.