PKE-UIC: A Prior Knowledge-Enhanced Underwater Image Captioning Model
Huanyu Li, Weibo Zhang, Zhuo Li, Peng Fei Ren · 2024
Underwater image captioning facilitates a transformation from "visual perception" to "semantic understanding" for underwater image analysis, providing substantial scientific and practical value for marine exploration and ecological monitoring.This paper proposes a prior knowledge-enhanced underwater image captioning model (PKE-UIC) based on a pure Transformer architecture.Firstly, we employ the pre-trained Swin Transformer as a feature extractor to derive grid features, which streamlines the training and testing processes and improves inference efficiency, making it particularly suitable for underwater scenes.Secondly, we propose a prior knowledge-enhanced attention encoder that facilitates the learning process of PKE-UIC by effectively integrating existing prior knowledge derived from samples within underwater image captioning datasets.This methodology enriches the features of the encoder, thereby improving its overall performance.Finally, we utilize a densely connected M2 decoder to fully exploit the output of the prior knowledge-enhanced attention features across all encoding layers during decoding, yielding more accurate caption generation for underwater images.Extensive comparative and ablation experiments validate the effectiveness of our proposed model.