"A Distorted Skull Lies in the Bottom Center..." Identifying Paintings from Text Descriptions
Anupam Guha, Mohit Iyyer, Jordan Lee Boyd-Graber · 2016
Most question answering systems use symbolic or text information.We present a dataset for a task that requires understanding descriptions of visual themes and their layout: identifying paintings from their descriptions.We annotate paintings with contour data, align regions with entity mentions from an ontology, and associate image regions with text spans from descriptions.A simple embedding-based method applied to text-to-image coreferences achieves state-of-the-art results on our task when paired with bipartite matching.The task is made all the more difficult by scarcity of training data.