Integrating Surface and Abstract Features for Robust Cross-Domain Chinese Word Segmentation
Xiaoqing Li, Kun Wang, Chengqing Zong, Keh‐Yih Su · International Conference on Computational Linguistics · 2012
Current character-based approaches are not robust for cross domain Chin ese word segmentation. In this paper, we alleviate this problem by deriving a novel enhanced ch aracter-based generative model with a new abstract aggregate candidate-feature, which indicates if th e given candidate prefers the corresponding position-tag of the longest dictionary matching wo rd. Since the distribution of the proposed feature is invariant across domains, our model thus possesses better generalization ability. Open tests on CIPS-SIGHAN-2010 show that the enhanced generative model achieves robust cross-domain performance for various OOV coverage rates and obtains the best performance on three out of four domains. The enhanced gen erative model is then further integrated with a discriminative model which also utilizes dictionary information . This integrated model is shown to be either superior or comparable to all other models repo rted in the literatur e on every domain of this task.