Using self-supervised word segmentation in Chinese information retrieval
Fuchun Peng, Xiangji Huang, Dale Schuurmans, Nick J. Cercone, Stephen E. Robertson · 2002
We propose a self-supervised word-segmentation technique for Chinese information retrieval. This method combines the advantages of traditional dictionary based approaches with character based approaches, while overcoming many of their shortcomings. Experiments on TREC data show comparable performance to both the dictionary based and the character based approaches. However, our method is language independent and unsupervised, which provides a promising avenue for constructing accurate multilingual information retrieval systems that are flexible and adaptive.