Efficient Word Lattice Generation for Joint Word Segmentation and POS Tagging in Japanese
Nobuhiro Kaji, Masaru Kitsuregawa · International Joint Conference on Natural Language Processing · 2013
This paper investigates the importance of a word lattice generation algorithm in joint word segmentation and POS tagging. We conducted experiments on three Japanese data sets to demonstrate that the previously proposed pruning-based algorithm is in fact not efficient enough, and that the pipeline algorithm, which is introduced in this paper, achieves considerable speedup without loss of accuracy. Moreover, the compactness of the lattice generated by the pipeline algorithm was investigated from both theoretical and empirical perspectives.