Unsupervised Chinese Word Segmentation Based on Minimum Information Content
Aoyuan Jiang, Dongchen Jiang, Xi Tang · 2024
Chinese word segmentation plays an important role in Chinese Natural Language Processing. In recent years, the supervised segmentation methods based on the neural network have received abundant attention and achieved lots of applications, but these methods require large-scale annotated corpora and long time for training. In this paper, we propose an unsupervised Chinese word segmentation method to tackle the problem of ambiguous segmentation. Experiment results show that our method has better performance and higher stability on different datasets, which renders it an effective segmentation tool for various Chinese NLP tasks.