A Neural Language Model with a Modified Attention Mechanism for Software Code
Xian Zhang, Kerong Ben · 2018
The language model, which has its roots in statistical natural language processing, has been shown to successfully capture the predictable regularities of source code, and help with many software tasks, such as code suggestion, code porting, and bug detection. However, learning long-range dependencies is still a big challenge at present for modeling programming language. In this paper, we explore deep learning techniques to strengthen it, with a novel neural language model for software code (NLM4Code) proposed. This model is mainly constructed by a recurrent neural network and augmented with a modified key-value-predict attention mechanism, which can effectively facilitate learning context dependencies of source code in a simpler way. To evaluate the effectiveness, five language models are selected as the baselines on a dataset of 2.03M lines of Java code. The experimental results show NLM4Code is much better than traditional n-gram models and outperforms a state-of-the-art method by 3.12% in perplexity index.