Rapid Algorithm of Chinese High-frequency Repeat Extraction Based on Hierarchical Pruning
Zhang Hai-ju · 2014
To extract high-frequency repeats from large-scale corpus,by using the hash table structure,this paper put forward a hierarchical pruning algorithm based on low-frequency character filtration and Cascade Pruning to filtrate lowfrequency strings and to reduce the times of I/O reading writing.On this basis,this paper employed the improved string sort algorithm,which can implement string sort in O(n)time complexity,to improve the efficiency of repeat extraction.According to the experimental data,it can be concluded that this algorithm is an effective method of repeat extraction,in which the relationship between the times of I/O reading writing and the scale of corpus is linear.The algorithm can effectively extract repeats from text corpus whose scale is much larger than that of the computer memory and can better suport the repeat-based applications such as new words identification,term extraction,etc.