Stochastic Korean Word-Spacing with Smoothing Using Korean Spelling Checker
Hyuk‐Chul Kwon, Mi-Young Kang, Sung‐Ja Choi · International Journal of Computer Processing Of Languages · 2004
Word-spacing errors, one of the most frequent errors in Korean, produce ambiguities in the lexical interpretation of parts of speech or render sentences including them incomprehensible. Resolving those errors is thus crucial in Korean language processing application domains. In this paper, we propose a stochastic Korean word-spacing system with smoothing using Korean Spelling Checker, which is equally robust for both inner data and external data. In order to cope with various problems of word-spacing, this study (a) presents a simple stochastic word-spacing system with only two parameters using the odds favoring the inner-spacing of a given syllable bigram as well as relative word ftequencies, and (b) endeavors to (i) remove noise from the training data and (ii) diminish training data-dependency by dynamically creating a candidate word with a longest-radix-selecting algorithm. The system thus becomes robust against unseen words and offers a similar performance for both inner data and external data: it obtained a 98.47% and a 97.78% precision in word-unit correction for the inner test data and the balanced external test data, respectively.