GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions
Ting-Yao Hsu, Chieh-Yang Huang, Ryan A. Rossi, Sungchul Kim, Clyde Lee Giles, Ting-Hao Huang · 2023
There is growing interest in systems that generate captions for scientific figures.However, assessing these systems' output poses a significant challenge.Human evaluation requires academic expertise and is costly, while automatic evaluation depends on often low-quality author-written captions.This paper investigates using large language models (LLMs) as a cost-effective, reference-free method for evaluating figure captions.We first constructed SCICAP-EVAL, 1 a human evaluation dataset that contains human judgments for 3,600 scientific figure captions, both original and machinemade, for 600 arXiv figures.We then prompted LLMs like GPT-4 and GPT-3 to score (1-6) each caption based on its potential to aid reader understanding, given relevant context such as figure-mentioning paragraphs.Results show that GPT-4, used as a zero-shot evaluator, outperformed all other models and even surpassed assessments made by Computer Science and Informatics undergraduates, achieving a Kendall correlation score of 0.401 with Ph.D. students' rankings.1 SCICAP-EVAL is accessible at https://github.com/ Crowd-AI-Lab/SciCap-Eval.