Joint Chinese word segmentation and punctuation prediction using deep recurrent neural network for social media data
Kui Wu, Xuancong Wang, Nina Zhou, Ai Ti Aw, Haizhou Li · 2015
In this work, we propose to jointly perform Chinese word segmentation (CWS) and punctuation prediction (PU) in a unified framework using deep recurrent neural network (DRNN). We further perform a comparative study among the joint frameworks, the isolated prediction and the pipeline methods that link the two tasks sequentially, on a social media corpus. Our experimental results show that joint models improve performance of CWS and affect PU marginally. We also study the effects of CWS and PU on Chinese-to-English machine translation (MT) quality by evaluating on a parallel social media corpus. It is shown that joint models are superior to the isolated prediction and the pipeline approaches.