New Words Detection Research for Programming Domain
Yongfeng Huang, Meng Liu, Xueyang Li · 2021
At present, there is no open domain vocabulary in the programming domain, which leads to many errors in Chinese word segmentation in programming domain. In order to promote the technical progress of word segmentation and other related tasks, the new words detection in the programming domain has important research significance. In this paper, the current phrase quality estimate method (ClassPhrase) for high-quality phrase extraction task is introduced into new words detection task. According to the existing domain vocabulary, we use the distant supervision method to generate labels, so that the label quality of the generated model training is higher, so as to train a better classification model to better complete the new words detection task. At the same time, the problem-solving reports are made public for the open data set lacking of text specification in the programming domain. The experimental results show that the method is effective in the programming domain. The F1_Score (F1) is 71.78%, which is better than the methods based on statistics and word embedding.