Attention Visualization of Gated Convolutional Neural Networks with Self Attention in Sentiment Analysis
Hidekazu Yanagimto, Kiyota Hashimoto, Makoto Okada · 2018
Deep learning is applied to many research topics; Natural Language Processing, Image Processing, and Acoustic Recognition. In deep learning, neural networks have a very complex and deep structure and it is difficult to discuss why they work well or not. So you have to take a trial-and-error to improve their performances. We develop a mechanism to show how neural networks predict final results and help you to design a new neural network architecture based on its prediction criteria. Speaking concrete, we visualize important features to predict the final results with an attentional mechanism. In this paper, we take up sentient analysis, which is one of natural language processing tasks. In image processing visualizing weights of a neural network is a major approach and you can obtain intuitive results; object outlines and object components. However, in natural language processing, the approach is not interpretable because a discriminate function constructed by a neural network is a complex and nonlinear one and it is very difficult to correlate weights and words in a text. We employ Gated Convolutional Neural Network (GCNN) and introduce a self-attention mechanism to understand how GCNN determines sentiment polarities from raw reviews. GCNN can simulate an n-gram model and the self-attention mechanism can make correspondence between weights of a neural network and words clear. In experiments, we used Amazon reviews and evaluated the performance of the proposed method. Especially, the proposed method was able to emphasize some words in the review to determine sentiment polarity. Moreover, when the prediction was wrong, we were able to understand why the proposed method made mistakes because we found what words the proposed method emphasized.