Sentence boundary detection of Uyghur based on rules and statistics
Tuergen Yibulayin · Computer Engineering and Applications Journal · 2010
Sentence boundary is an important initial task for many natural language processing applications,such as part-of-speech tagging and parsing etc.This paper proposes an automatic sentence boundary detection method of Uyghur based on rules and statistic.Firstly,the paragraph detecting algorithm classifies the ambiguous and unambiguous paragraph.In the second step,the rule based sentence boundary detector process the unambiguous paragraphs.Finally,the maximum entropy based sentence boundary detecting model identifies the ambiguous paragraph sentences.This method improves robustness of the method by making plenty use of rule to reduce the failure of the ME model to identify the unambiguous paragraphs which can be attributed to the sparsity of the training data used and the ME model to resolve ambiguity,the recall of this method reaches 98.77%.