Thai Sentence-Breaking for Large-Scale SMT
Glenn Slayden, Mei-Yuh Hwang, Lee Schwartz · 2010
Thai language text presents challenges for integration into large-scale multilanguage statistical machine translation (SMT) systems, largely stemming from the nominal lack of punctuation and inter-word space. For Thai sentence breaking, we describe a monolingual maximum entropy classifier with features that may be applicable to other languages such as Arabic, Khmer and Lao. We apply this sentence breaker to our largevocabulary, general-purpose, bidirectional Thai-English SMT system, and achieve BLEU scores of around 0.20, reaching our threshold of releasing it as a free online service. 1