Normalized Accessor Variety Combined with Conditional Random Fields in Chinese Word Segmentation
Saike He, Taozheng Zhang, Xue Bai, Xiaojie Wang, Yuan Dong · Recent Advances in Natural Language Processing · 2009
The word is the basic unit in natural language processing (NLP), as it is at the lexical level upon which further processing rests. The lack of word delimiters such as spaces in Chinese texts makes Chinese word segmentation (CWS) an interesting while challenging issue. This paper describes the in-depth research following our participation in the fourth International Chinese Language Processing Bakeoff 1 . Originally, we incorporate unsupervised segmentation into Conditional Random Fields (CRFs) in the purpose of dealing with unknown words. Normalization is delicately involved in order to cater to problem of small data size. Experiments on CWS corpora from Bakeoff-4 present comparable results with state-of-the-art performance.