Design of natural language model based on BiGRU and attention mechanism
Di Luan, Yang Xiushuang, Ling Xie · 2021
This paper implements a natural language model design based on bidirectional GRU (BiGRU) and attention mechanism. Natural language model is an important part of natural language processing and a typical prediction problem. BiGRU is a bidirectional GRU, an improved LSTM network. It can not only make effective use of context information, but also reduce training parameters and improve efficiency. The attention mechanism focuses on the most relevant information according to the weight distribution. The algorithm is implemented based on Tensorflow platform. Words are expressed as dense vectors by word2vec method. In order to prevent over fitting, Dropout mechanism is superimposed in the model. Finally, Perplexity is used to evaluate the performance of the model. Then the BiLSTM model is also tested. The results show that BiGRU model has better performance in Perplexity and learning efficiency.