A news image captioning approach based on multimodal pointer‐generator network
Jingqiang Chen, Hai Zhuge · Concurrency and Computation Practice and Experience · 2020
Summary News image captioning aims to generate captions or descriptions for news images automatically, serving as draft captions for creating news image captions manually. News image captions are different from generic captions as news image captions contain more detailed information such as entity names and events. Therefore, both images on news and the accompanying text are the source of generating caption of news image. Pointer‐generator network is a neural method defined for text summarization. This article proposes the Multimodal pointer‐generation network by incorporating visual information into the original network for news image captioning. The multimodal attention mechanism is proposed by splitting attention into visual attention paid to the image and textual attention paid to the text. The multimodal pointer mechanism is proposed by using both textual attention and visual attention to compute pointer distributions, where visual attention is first transformed into textual attention via the word‐image relationships. The multimodal coverage mechanism is defined to reduce repetitions of attentions or repetitions of pointer distributions. Experiments on theDailyMailtest dataset and the out‐of‐domainBBCtest dataset show that the proposed model outperforms the original pointer‐generator network, the generic image captioning method, the extractive news image captioning method, and theLDA‐based method accordingBLEU,METEOR, andROUGL‐Levaluations. Experiments also show that the proposed multimodal coverage mechanisms can improve the model, and that transforming visual attention to pointer distributions can improve the model.