Exploring Visual Attention Mechanism for Scene Understanding in Image Captioning
Zongjian Zhang · UTS ePRESS (University of Technology Sydney) · 2020
Scene understanding is a high-level computer vision research task that requires multiple fundamental vision tasks on different visual elements.Image captioning is a typical scene understanding task that understands salient visual contents in a real-world scene of an image and then automatically describes its understandings via a natural language sentence.This thesis concentrates on exploring visual attention mechanism for the scene understanding in image captioning, aiming to achieve a comprehensive multimodal and transparent scene understanding ability.Specifically, four problems are studied to enhance the visual attention mechanism to fine-grained spatial level, to a comprehensive semantic level, and to a high-level ability of attending to visual relationships for interaction words.Firstly, this thesis proposes a fine-grained and semantic-guided visual region attention model based on a novel Fully Convolutional Network (FCN)-Long Short Term Memory (LSTM) framework.It can attend to both object and "stuff" regions at a fine-grained grid-wise resolution and only focuses on the principal object information in each grid cell.In addition, grid-wise semantic labels are introduced to provide semantic guidance to ensure that related visual regions in different grid cells are correlated to each other.Moreover, an additional semantic context can be summarized from these textual semantic labels.Secondly, this thesis proposes a novel high-resolution FCN encoder is used with First of all, I would like to express my deepest gratitude to my principal supervisor, A/Prof.Qiang Wu.Without his guidance, this thesis would not have been accomplished with high quality, and I would not have achieved qualified PhD research outcomes.With abundant knowledge and the insightful and rigorous way of thinking about research problems, he patiently guided me through many research challenges in this valuable journey of PhD study.Besides this degree itself, I've learned the methodology of exporing and solving a new problem, which I believe is more important for my future work and life.Moreover, he is also a great friend that cares about my personal life and offers helps when I face problems.I feel very lucky to finish my PhD under his supervision and can't imagine a better supervisor.Deeply appreciate it