Language Models can Evaluate Themselves via Probability Discrepancy
Tingyu Xia, Bowen Yu, Yuan Wu, Yi Chang, Chang Zhou · 2024
In this paper, we initiate our discussion by demonstrating how Large Language Models (LLMs), when tasked with responding to queries, display a more even probability distribution in their answers if they are more adept, as opposed to their less skilled counterparts.Expanding on this foundational insight, we propose a new self-evaluation method ProbDiff for assessing the efficacy of various LLMs.This approach obviates the necessity for an additional evaluation model or the dependence on external, proprietary models like GPT-4 for judgment.It uniquely utilizes the LLMs being tested to compute the probability discrepancy between the initial response and its revised versions.A higher discrepancy for a given query between two LLMs indicates a relatively weaker capability.Our findings reveal that ProbDiff achieves results on par with those obtained from evaluations based on GPT-4, spanning a range of scenarios that include natural language generation (NLG) tasks such as translation, summarization, and our proposed Xiaohongshu blog writing task, and benchmarks for LLM evaluation like AlignBench, MT-Bench, and AlpacaEval, across LLMs of varying magnitudes.The code is available at https:// github.com/xiatingyu/ProbDiff.