Juman++: A Morphological Analysis Toolkit for Scriptio Continua
Arseny Tolmachev, Daisuke Kawahara, Sadao Kurohashi · 2018
We present a three-part toolkit for developing morphological analyzers for languages without natural word boundaries.The first part is a lattice-based morphological analysis library that uses a combination of linear and recurrent neural net language models for analysis.The other parts are a tool for exposing problems in the trained model and a partial annotation tool.Our morphological analyzer for Japanese achieves new SOTA on Jumandic-based corpora while being 250 times faster than the previous one.We also perform a small experiment and quantitive analysis of using our toolkit.