CRFs-Based Chinese Word Segmentation for Micro-Blog with Small-Scale Data
Longyue Wang, Derek F. Wong, Lidia Sam Chao, Junwen Xing · 2012
In this paper, we proposed a Chinese word segmentation model for micro-blog text. Alt-hough Conditional Random Fields (CRFs) models have been presented to deal with word segmentation, this is still the first time to apply it for the segmentation in the domain of Chi-nese micro-blog. Different from the genres of common articles, micro-blog has gradually be-come a new literary with the development of Internet. However, the unavailable of micro-blog training data has been the obstacle to de-velop a good segmenter based on trainable models. Considering the linguistic characteris-tics of the text, we proposed some methods to make the CRFs models suitable for segmenta-tion in the domain of micro-blog. Several ex-periments have been conducted with different settings and then an optimal tagging method and feature templates have been designed. The proposed model has been implemented for the Second CIPS-SIGHAN Joint Conference on Chinese Language Processing Bakeoff (Bakeoff-2012) and achieves a very high F-measure of 93.38 % within the test set of 5,000 micro-blog sentences. One of our main contri-butions is the online version of toolkit 1, which provides segmentation service for Chinese mi-cro-blog text. 1