Semantic Tensor Product for Image Captioning
Chiranjib Sur, Pei Liu, Yingjie Zhou, Dapeng Oliver Wu · 2019
In this paper, we proposed a Semantic Tensor Product (STP) model aimed at finding a more grammatically correct caption for a given image. We tried two kinds of ways to integrate semantic concepts (i.e. tags) into Tensor Product Representation (TPR) in the process of image caption generation, where Tensor Product Representation has the advantage of decoding a role embedding using a corresponding unbinding vector in any position directly, while semantic concepts could provide crucial clues for internal TPR prediction. We qualitatively analyze the two proposed STP based models, and quantitatively evaluate them on benchmark dataset: MSCOCO. Experiment result shows that our results outperform the other models, and generated captions are more natural.