Adaptive Method for Chinese New Word Identification Based on Multi-features
Zhiyong Luo · Journal of Beijing University of Technology · 2007
To improve the performance of new word identification in Chinese word segment,the authors pro- pose an adaptive method for Chinese new word identification based on multi-feature method for off line corpus processing,in which many features,including context-entropy,likelihood ratios,frequency ratio against background corpus and boundary-verification with basic segmentation are introduced to evaluate the candidate words.And all of the features are integrated into an adaptive SVM classifier.Candidate new words are extracted efficiently on PAT-Array with much less space overhead and arbitrary n-gram words can be identi- fied by the method.The results show that the method can run fast upon new word identification and save much memory.