Semantic Tuples for Evaluation of Image to Sentence Generation
Lily D. Ellebracht, Arnau Ramisa, Pranava Madhyastha, Jose Cordero-Rama, Francesc Moreno-Noguer, Ariadna Quattoni · 2015
The automatic generation of image captions has received considerable attention.The problem of evaluating caption generation systems, though, has not been that much explored.We propose a novel evaluation approach based on comparing the underlying visual semantics of the candidate and ground-truth captions.With this goal in mind we have defined a semantic representation for visually descriptive language and have augmented a subset of the Flickr-8K dataset with semantic annotations.Our evaluation metric (BAST) can be used not only to compare systems but also to do error analysis and get a better understanding of the type of mistakes a system does.To compute BAST we need to predict the semantic representation for the automatically generated captions.We use the Flickr-ST dataset to train classifiers that predict STs so that evaluation can be fully automated 1 .