Domain Neural Chinese Word Segmentation with Mutual Information and Entropy
Jun Wang, Ge Bin, Chunhui He · 2019
Chinese word segmentation (CWS) is an important basic task for NLP. However, the word segmentation model trained by the generic domain corpus has a significant decline in performance in the word segmentation task oriented to the specific domain. Aiming at the features of domain segmentation, this paper using domain corpus as the training samples, and proposed combined with the terminology dictionary, new word detection and Bi-LSTM-CRF segmentation method to improve the problem of out-of-vocabulary (OOV). The word segmentation experiment was carried out on the corpus of the automotive domain. The results show that the precision and recall of the word segmentation have reached 0.95, and the value of F1 also achieved 0.95, and they are better than state-of-the-art method. This method can also be combined with N-gram and chi-square statistic to further improve the recognition accuracy of OOV.