Online Japanese Unknown Morpheme Detection using Orthographic Variation
Yugo Murawaki, Sadao Kurohashi · 2010
To solve the unknown morpheme problem in Japanese morphological analysis, we previously proposed a novel framework of online unknown morpheme acquisition and its implementation.This framework poses a previously unexplored problem, online unknown morpheme detection.Online unknown morpheme detection is a task of finding morphemes in each sentence that are not listed in a given lexicon.Unlike in English, it is a non-trivial task because Japanese does not delimit words by white space.We first present a baseline method that simply uses the output of the morphological analyzer.We then show that it fails to detect some unknown morphemes because they are over-segmented into shorter registered morphemes.To cope with this problem, we present a simple solution, the use of orthographic variation of Japanese.Under the assumption that orthographic variants behave similarly, each over-segmentation candidate is checked against its counterparts.Experiments show that the proposed method improves the recall of detection and contributes to improving unknown morpheme acquisition.