JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models
Yuiga Wada, Kanta Kaneda, Komei Sugiura · 2023
Image captioning studies heavily rely on automatic evaluation metrics such as BLEU and METEOR.However, such n-gram-based metrics have been shown to correlate poorly with human evaluation, leading to the proposal of alternative metrics such as SPICE for English; however, no equivalent metrics have been established for other languages.Therefore, in this study, we propose an automatic evaluation metric called JaSPICE, which evaluates Japanese captions based on scene graphs.The proposed method generates a scene graph from dependencies and the predicate-argument structure, and extends the graph using synonyms.We conducted experiments employing 10 image captioning models trained on STAIR Captions and PFN-PIC and constructed the Shichimi dataset, which contains 103,170 human evaluations.The results showed that our metric outperformed the baseline metrics for the correlation coefficient with the human evaluation.