Chinese Word Segmentation Based on Deep Learning
Mengge Wang, Xiaoge Li, Zheng Wei, Shuting Zhi, Haoyue Wang · 2018
Chinese word segmentation is a fundamental task in the field of Chinese Natural Language Processing. In this paper, we propose a series of neural network architectures by combining Long Short-Term Memory Neural Network (LSTM) with Conditional Random Field (CRF). Firstly, we use the Skip-Gram model to obtain character embeddings from a raw corpus (without word delimiters). Then, we treat these embeddings as inputs of LSTM model to compute their context representation vectors. Finally, these representation vectors are applied to the CRF layer for Chinese word segmentation. Experiments on the corpus of the 2014 people's daily and Chinese Computing (NLPCC2015) Weibo texts segmentation task show that our models outperforms the previous machine learning methods.