Sub-character Neural Language Modelling in Japanese
Viet Anh Nguyen, Julian Brooke, Timothy J. Baldwin · 2017
In East Asian languages such as Japanese and Chinese, the semantics of a character are (somewhat) reflected in its sub-character elements.This paper examines the effect of using subcharacters for language modeling in Japanese.This is achieved by decomposing characters according to a range of character decomposition datasets, and training a neural language model over variously decomposed character representations.Our results indicate that language modelling can be improved through the inclusion of subcharacters, though this result depends on a good choice of decomposition dataset and the appropriate granularity of decomposition.