Ad Lingua: Text Classification Improves Symbolism Prediction in Image Advertisements
Andrey Savchenko, Anton Alekseev, Sejeong Kwon, Elena Viktorovna Tutubalina, Evgeny V. Myasnikov, Sergey Nikolenko · 2020
Understanding image advertisements is a challenging task, often requiring non-literal interpretation.We argue that standard image-based predictions are insufficient for symbolism prediction.Following the intuition that texts and images are complementary in advertising, we introduce a multimodal ensemble of a state of the art image-based classifier, a classifier based on an object detection architecture, and a fine-tuned language model applied to texts extracted from ads by OCR.The resulting system establishes a new state of the art in symbolism prediction.