Chinese Unknown Word Recognition Using Improved Conditional Random Fields
Yisu Xu, Xuan Wang, Buzhou Tang, Xiaolong Wang · 2008
Unknown word recognition is a very important problem in natural language processing. It has a great influence on the performance of dictionary construction and word segmentation. This paper introduces two methods to improve the effect of Chinese unknown word recognition by using Conditional Random Fields: the rough label of the characters and the N-best listing. The CRF with the two methods proposed by this paper can increase recall rate of out-of-vocabulary (ROOV) against original CRF model by 15% which is the key point when doing unknown word recognition. It has the same result as the highest recall rate of OOV in Sighan Bakeoff 2005 close test on Peking University (PKU) corpora, however, a much higher recall rate of in-vocabulary (RIV) than others.