Effective Subsequence-Based Tagging for Chinese Word Segmentation
Hai Yan Zhao, Chunyu Kit · Zhongwen xinxi xuebao · 2007
The research of automatic Chinese word segmentation has been advancing rapidly in recent years, espe- cially since the First International Chinese Word Segmentation Bakeoff held in 2003. In particular, character-based tagging has claimed a great success in this field. In this paper, we attempt to generalize this method to subse- quence-based tagging. Our goal is to find longer tagging units through a reliable algorithm. We propose a two-step framework to serve this purpose. In the first step, an iterative maximum matching filtering algorithm is applied to obtain an effective subsequence lexicon, while in the second step, a bi-lexicon based maximum matching algo- rithm is used for identifying subsequence units. The effectiveness of this approach is verified by our experiments using two closed test data sets from Bakeoff-2005.