Text-Enriched Representations for News Image Classification
Elias Moons, Tinne Tuytelaars, Marie‐Francine Moens · 2018
Images have a prominent role in the communication of news on the Web. We propose a novel method for image classification with subject categories when limited annotated images are available for training the classifier. A neural network based encoder learns image representations from paired news images and their texts. Once trained, this encoder transforms any image to a text-enriched representation of the image, which is then used as input for the classifier that categorizes an image according to its subject category. We have trained classifiers with different amounts of annotated images and found that the image classifier that uses the text-enriched image representations outperforms a baseline model that only uses image features especially in cases with limited training examples.