Spectral Graph-Based Method of Multimodal Word Embedding
Kazuki Fukui, Takamasa Oshikiri, Hidetoshi Shimodaira · 2017
In this paper, we propose a novel method for multimodal word embedding, which exploit a generalized framework of multiview spectral graph embedding to take into account visual appearances or scenes denoted by words in a corpus.We evaluated our method through word similarity tasks and a concept-to-image search task, having found that it provides word representations that reflect visual information, while somewhat trading-off the performance on the word similarity tasks.Moreover, we demonstrate that our method captures multimodal linguistic regularities, which enable recovering relational similarities between words and images by vector arithmetic.