A Text Editing Approach to Joint Japanese Word Segmentation, POS Tagging, and Lexical Normalization
Shohei Higashiyama, Masao Utiyama, Taro Watanabe, Eiichiro Sumita · 2021
Lexical normalization, in addition to word segmentation and part-of-speech tagging, is a fundamental task for Japanese user-generated text processing.In this paper, we propose a text editing model to solve the three task jointly and methods of pseudo-labeled data generation to overcome the problem of data deficiency.Our experiments showed that the proposed model achieved better normalization performance when trained on more diverse pseudo-labeled data.