Research on Deep Learning Based Code Generation from Natural Language Description
Jiaqi Zhu, Mingzhu Shen · 2020
With significant advances in deep learning, the code generation from natural language description has become a prevailing research. Existing researches demonstrate that these methods have achieved high BLEU values. However, the data sets used in the existing researches lack diversity, and they usually use BLEU as the only evaluation metric. To overcome these limitations, in this paper, we crawled a data set that is more suitable for code generation from the online judge system, and re-run the existing code generation models on this data set. We evaluate the generated code from five aspects: lexical similarity, tree similarity, syntactic legality, semantic legality, and functional correctness. This study provides a deeper analysis of the performance of existing code generation methods.