Discriminative Stem-Affix Segmentation for Directed-Graph-Based Mongolian Lexical Analyzer
Jinxing Wu · Zhongwen xinxi xuebao · 2011
In Mongolian lexical analysis,the directed-graph-based model achieves high performance.This model uses a directed-graph architecture to describe the probabilistic relationship of stems and affixes,thus to determine the best segmented and tagged candidate for each word according to the context.Therefore,it is essential for a directed-graph-based analyzer to enumerate all legal segmented and tagged candidates for each word.This paper proposes a novel stem-affix segmentation model based on discriminative classification method for Mongolian lexical analysis.Compared with the enumeration strategy based on the stem-and affix sets,this method shows better generalization ability for the words with unknown stems.Using the 3rd-level annotated corpus with about 200000 words as the training data,the directed-graph-based lexical analyzer with discriminative stem-affix segmentation module achieves further 7% improvement on F1 measure(with unknown stems considered).