CapOnImage: Context-driven Dense-Captioning on Image
Yiqi Gao, Xinglin Hou, Yuanmeng Zhang, Tiezheng Ge, Yuning Jiang, Peng Wang · 2022
Existing image captioning systems are dedicated to generating narrative captions for images, which are spatially detached from the image in presentation.However, texts can also be used as decorations on the image to highlight the key points and increase the attractiveness of images.In this work, we introduce a new task called captioning on image (CapOn-Image) 1 , which aims to generate dense captions at different locations of the image based on contextual information.For this new task, we introduce a large-scale benchmark called CapOn-Image2M, which contains 2.1 million product images, each with an average of 4.8 spatially localized captions.To fully exploit the surrounding visual context to generate the most suitable caption for each location, we propose a multimodal pre-training model with multi-level pretraining tasks that progressively learn the correspondence between texts and image locations from easy to hard.To avoid generating redundant captions for nearby locations, we further enhance the location embedding with neighbor locations .Compared with other image captioning model variants, our model achieves the best results in both captioning accuracy and diversity aspects.