Which is better? Exploring Prompting Strategy For LLM-based Metrics
JoongHoon Kim, Sangmin Lee, Seung Hun Han, So-Yun Park, Jiyoon Lee, Kiyoon Jeong, Pilsung Kang · 2023
This paper describes the DSBA submissions to the Prompting Large Language Models as Explainable Metrics shared task, where systems were submitted to two tracks: small and large summarization tracks.With advanced Large Language Models (LLMs) such as GPT-4, evaluating the quality of Natural Language Generation (NLG) has become increasingly paramount.Traditional similarity-based metrics such as BLEU and ROUGE have shown to misalign with human evaluation and are illsuited for open-ended generation tasks.To address this issue, we explore the potential capability of LLM-based metrics, especially leveraging open-source LLMs.In this study, wide range of prompts and prompting techniques are systematically analyzed with three approaches: prompting strategy, score aggregation, and explainability.Our research focuses on formulating effective prompt templates, determining the granularity of NLG quality scores and assessing the impact of in-context examples on LLM-based evaluation.Furthermore, three aggregation strategies are compared to identify the most reliable method for aggregating NLG quality scores.To examine explainability, we devise a strategy that generates rationales for the scores and analyzes the characteristics of the explanation produced by the open-source LLMs.Extensive experiments provide insights regarding evaluation capabilities of open-source LLMs and suggest effective prompting strategies. 1