Evaluating and improving transformers pre-trained on ASTs for Code Completion
Marcel Ochs, Krishna Narasimhan, Mira Mezini · 2023
Automatic code completion is one of the most popular developer assistance features which is usually performed using static program analysis. But, studies have shown that static analysis can be riddled with false positives. Another issue with static-analysis based code completion is that the recommendation does not take into account the history/ context in which the software is operating and rely on type/ alphabetical ordering of suggestions. A recent development that has shown to be promising in this direction is the use of language models such as transformers that are trained on real-world code from Github to provide context sensitive, accurate code completion suggestions. Studies on transformer-based code completion have shown that such restrictions can be leveraged; i.e; training transformers on structural representations of code (specifically ASTs) could have a positive impact on the accuracy of code completion. To this end, the work by Kim et al. implemented and evaluated TravTrans which is based on the powerful text prediction language model GPT-2 by training on abstract syntax trees instead of treating code as plain texts. Using alternative source code representation such as AST provides the already potent language model with an additional layer of program semantic-awareness. But, TravTrans has adapted several rigid choices regarding various components of the transformer architecture like embedding sizes, sliding windows etc. TravTrans also suffers from issues related to running out of vocabulary. In this paper, we reproduce the TravTrans model and perform a deeper, fine-grained analysis of the impact of various architectural and code-level settings on the prediction. As a result of our fine-grained analysis, we also identify several aspects that need improvements like the fact that the model performs particularly poorly with code involving dictionaries and lists. We also offer solutions to a few of the issues like the out-of-vocabulary issue. Finally, our results motivates the need for a customizable-transformer architecture for coding tasks.