Semantic Image Profiling for Historic Events
Jia Chen, Qin Jin, Yifan Xiong · 2016
Automatically generating image profiles for historic events is desired for history knowledge preservation and curation. However, a simple profile with groups of related images lacks explicit semantic information, such as which images correspond to which aspects of the event. In this paper, we propose to add explicit semantic information to image profiling by linking images in the profile with related phrases in the event description. We measure the relevance of an image-phrase pair via a real-valued matching score. We exploit instance-wise ranking loss function to learn the matching score and we deal with two challenges: 1) how to automatically generate labeled positive data: we leverage out-of-domain labeled datasets to generate pseudo positive in-domain labels and propose a new algorithm (WIL4PPL) to robustly learn the model from the noisy pseudo positive labels; 2) how to automatically generate negative data: we propose a negative set generation algorithm to guide the model in learning which phrases and images to distinguish. We compare our model to three baselines and conduct detailed analysis and case studies to verify the quality of learnt semantic information. The extensive experiment results show the effectiveness of our proposed algorithms which significantly outperform the baselines.