VeriScore: Evaluating the factuality of verifiable claims in long-form text generation
Yixiao Song, Yekyung Kim, Mohit Iyyer · 2024
Existing metrics for evaluating the factuality of long-form text, such as FACTSCORE (Min et al., 2023) and SAFE (Wei et al., 2024), decompose an input text into "atomic claims" and verify each against a knowledge base like Wikipedia.These metrics are not suitable for most generation tasks because they assume that every claim is verifiable (i.e., can plausibly be proven true or false).We address this issue with VERISCORE, 1 a metric for evaluating factuality in diverse long-form generation tasks that contain both verifiable and unverifiable content.VERISCORE can be effectively implemented with either closed or fine-tuned openweight language models.Human evaluation confirms that VERISCORE's extracted claims are more sensible than those from competing methods across eight different long-form tasks.We use VERISCORE to evaluate generations from 16 different models across multiple longform tasks and find that while GPT-4o is the best-performing model overall, open-weight models such as Mixtral-8 × 22 are closing the gap.We show that an LM's VERISCORE on one task (e.g., biography generation) does not necessarily correlate to its VERISCORE on a different task (e.g., long-form QA), highlighting the need for expanding factuality evaluation across tasks with varying fact density.Claim 1: Beetroot gets its red color from betacyanin.Claim 2: Betacyanins are a type of anthocyanin.Claim 3: Anthocyanins are water-soluble pigments.Claim 4: Anthocyanins are commonly found in various fruits and vegetables.Claim 1: Beetroot gets its red color from betacyanin.Claim 2: Betacyanin is a special helper that gives beetroot its beautiful red color.Claim 3: Betacyanins are a type of anthocyanin.Claim 4: Anthocyanins are water-soluble pigments.Claim 5: Anthocyanins are commonly found in various fruits and vegetables Claim 6: Betacyanin is like a superhero cape.