Error-Driven Learning of Chinese Word Segmentation
Julia Hockenmaier, C. Brew · Institutional Repositories DataBase (IRDB) · 1998
Palmer ([4]) demonstrated how Brill's Transformation-based Error-Driven Learning can be applied to word segmentation in various languages. We present experimental results which show that such algorithms can achieve satisfactory performance even with a a very simple initial state annotator We also present two preliminary studies, which suggest that even higher performancemight be achieved if simple morphological information is available to the system, and that segmentation performance might actually be improved by combining segmentation with rudimentary part-of-speech tagging. 1 Introduction Chinese word segmentation is an interesting, but difficult problem. The difficulties include the following: - "word" is not a very well-defined concept in the context of Chinese: linguists do not have generally accepted guidelines, and in experiments native speakers show only about 75 % agreement on the "correct" segmentation. - Even if we have guidelines, the problem does not become trivial. The b...