Apply Normalized Accessor Variety in Chinese Word Segmentation
Saike He, Xiaojie Wang, Dong Yuan, Taozheng Zhang, Xue Bai · Zhongwen xinxi xuebao · 2010
This paper proposes a method combining supervised learning with unsupervised method to conduct Chinese word segmentation(CWS),which incorporates the Accessor Variety(AV) into the Conditional Random Fields(CRFs).To solve the flaw in Accessor Variety(AV) when dealing with limited training data,normalization is introduced to alleviate the fluctuation in the AV value in the phrase of unsupervised segmentation.Experiments on the Bakeoff-4 data indicate that normalized Accessor Variety is effective both for close and open tracks..