This is not correct! Negation-aware Evaluation of Language Generation Systems
Miriam Anschütz, Diego Miguel Lozano, Georg Groh · 2023
Large language models underestimate the impact of negations on how much they change the meaning of a sentence.Therefore, learned evaluation metrics based on these models are insensitive to negations.In this paper, we propose NegBLEURT, a negation-aware version of the BLEURT evaluation metric.For that, we designed a rule-based sentence negation tool and used it to create the CANNOT negation evaluation dataset.Based on this dataset, we fine-tuned a sentence transformer and an evaluation metric to improve their negation sensitivity.Evaluating these models on existing benchmarks shows that our fine-tuned models outperform existing metrics on the negated sentences by far while preserving their base models' performances on other perturbations.