Word Segmentation of Chinese Text with Multiple Hybrid Methods
Zhongjian Wang, Jun Xu, Kenji Araki, Koji Tochinai · 2009
To deal with unknown word and segmentation ambiguity, segmentation rules and tri-gram was used in inductive learning method. Rules were used for elementary segmentation and for better processing effectiveness in following steps. Those rules were acquired by manual labor through analyzing a tagged corpus. Inductive learning method recognized, extracted the unknown words from segmentation text recursively. The tri-gram model was used to deal with segmentation ambiguity, to select the better segmentation candidate by calculating a sentence probability. Experimental results indicated that unknown words processing and segmentation error were improved.