Algorithm for Removing Similar Sentence on Parallel Corpus
Liu Lian-fang · Journal of Guangxi Academy of Sciences · 2009
The similarity of Chinese sentence is classified and duplicated sentence is removed.Sentence similarity depends on similarity of unitary factor and partial factor.According to the idea of KMP's jump,the simular KMP in chinese sentence is used.The experiment results show that the algorithm is effective,the recall rate of duplicate removal reach 94%,and the precision rate reach 84% in large scale testing.