Parameters Correlation and Optimization in Text Similarity Measurement
Zuping Zhang, Xin Yan Xu, Jun Bo Long, Xinpan Yuan · Journal of Chinese Computer Systems · 2011
Parameters in text similarity measurement such as similarity threshold,precision,recall rate,size of shingle moving window,shingle weighted coefficient and text attributes are interrelated and their relationship are complicated.Based on the analysis of parameter correlations and practical requirements,we suggest optimized parameter configurations and design a similarity measurement algorithm to adjust the similarity threshold to the text length contribute.The algorithm is applied to the text similarity analysis of 7378 proposals for some fund in 2009.The results demonstrate that,no matter the text length is long or short,the algorithm is so efficient that the precision and recall rate are both higher than 95%.