Towards Better Evaluation of Instruction-Following: A Case-Study in Summarization
Ondrej Skopek, Rahul Aralikatte, Sian Gooding, Victor Cărbune · 2023
Despite recent advances, evaluating how well large language models (LLMs) follow user instructions remains an open problem.While evaluation methods of language models have seen a rise in prompt-based approaches, limited work on the correctness of these methods has been conducted.In this work, we perform a meta-evaluation of a variety of metrics to quantify how accurately they measure the instruction-following abilities of LLMs.Our investigation is performed on grounded query-based summarization by collecting a new short-form, real-world dataset riSum, containing 300 document-instruction pairs with 3 answers each.All 900 answers are rated by 3 human annotators.Using riSum, we analyze the agreement between evaluation methods and human judgment.Finally, we propose new LLMbased reference-free evaluation methods that improve upon established baselines and perform on par with costly reference-based metrics that require high-quality summaries.