CHINESE LEXICAL ANALYSIS BASED ON HYBRID MMSM MODEL
Xiao Na Sun, Degen Huang, Fuji Ren · International journal of innovative computing, information & control · 2009
In this paper, we describe a scheme for Chinese word segmentation and POS tagging which integrates the character-based and word-based information in the directed graph generated by the MMSM model. Word-level information is effective for analysis of known words, while character-level information is useful for analysis of unknown words. A Hidden semi-CRF model is proposed for the unknown words detection and POS tagging. The proposed Hidden semi-CRF has two state chains with unequal states which can perform segmentation and POS tagging of unknown words simultaneously. The hybrid model was evaluated using the test data from SIGHAN-6 and achieved higher F-score than the stage-of-the-art models.