An integrated approach for Chinese word segmentation
Guohong Fu, Kang Kwong Luke · Institutional Repositories DataBase (IRDB) · 2003
This paper presents an integrated approach for Chinese word segmentation, which can perform disambiguation and unknown word identification simultaneously on the input.In this work, a hybrid model is used to score known word candidates and unknown word candidates equally by incorporating the modified word-formation models (viz.word-juncture models and wordformation patterns) into word bigram models, with which different types of features are statistically computed and combined for this integrated segmentation, including internal wordformation power of components in a word, affinity relations between these components and the external contextual information.To enhance the precision and avoid the problem of combination explosion in word candidate construction, a filter algorithm is also given to block ineligible unknown word candidates.In this way, ambiguity and unknown word can be resolved effectively.The results of our experiment on Peking University corpus show that the integrated approach outperforms the other two-stage methods under discussion.