Active Learning in Chinese Word Segmentation Based on Multigram Language Model
Chong Feng, Zhaoxiong Chen · Zhongwen xinxi xuebao · 2006
Word segmentation is a fundamental task in Chinese processing.To solve the difficulties of traditional methods in coping with various application domains and evolutive language phenomena,this paper adopts an unsupervised learning framework,using EM algorithm to train the n-multigram language model.A new certainty-based active learning segmentation algorithm is proposed,which combine labeled data with unlabeled data together to optimize language model.In experiments it outperforms other unsupervised word segmentation algorithms.