Context-Based Chinese Word Segmentation using SVM Machine-Learning Algorithm without Dictionary Support
Chia-ming Lee, Chien‐Kang Huang · International Joint Conference on Natural Language Processing · 2013
This paper presents a new machine-learning Chinese word segmentation (CWS) approach, which defines CWS as a break-point classification problem; the break point is the boundary of two subsequent words. Further, this paper exploits a support vector machine (SVM) classifier, which learns the segmentation rules of the Chinese language from a context model of break points in a corpus. Additionally, we have designed an effective feature set for building the context model, and a systematic approach for creating the positive and negative samples used for training the classifier. Unlike the traditional approach, which requires the assistance of large-scale known information sources such as dictionaries or linguistic tagging, the proposed approach selects the most frequent words in the corpus as the learning sources. In this way, CWS is able to execute in any novel corpus without proper assistance sources. According to our experimental results, the proposed approach can achieve a competitive result compared with the Chinese knowledge and information processing (CKIP) system from Academia Sinica.