Chinese Tweets Segmentation based on Morphemes
Chaoyue Wang, Guohong Fu · 2012
Chinese tweets segmentation is a critical prob-lem in natural language processing area. While segmentation of in-vocabulary words is well studied to date, few research findings are yet available concerning the prediction of new words on twitter. In this paper, we attempt to exploit multiple features for segmenting tweets in real text. To this end, we first take morpheme as the basic component units of Chinese words and thus investigate the rela-tionship between Chinese new words and their internal morphological structures. Then, we explore both word internal cues and word ex-ternal contextual features, and combine them for segmentation of Chinese new words using conditional random field. Our experimental re-sults show that the incorporation of multiple features, especially the word-internal morpho-logical features is of great value to Chinese tweets segmentation. 1