Text Similarity Calculation Method Based on Hybrid Model of LDA and TF-IDF
Jiangyao Wang, Wenhua Xu, Wenhao Yan, Caixia Li · 2019
The traditional TF-IDF-based text similarity calculation model uses statistical methods to map text to the keyword vector space and convert the similarity of text into the distance between text vectors. Such methods have problems such as high computational dimensions, sparse data, and inability to take advantage of the semantic information contained in the text itself, so the results obtained are not as similar as the physical text. The text similarity model based on the topic model changes the traditional spatial similarity of keyword vectors, and can fully utilize the semantic information contained in the text itself. But this approach ignores the effect of words on text semantic representations with different weights. In the process of converting text into topic feature space, valuable information is lost. In view of the above problems, this paper proposes a text similarity hybrid model (L-THM) integrating LDA and TF-IDF for calculating text similarity. The model uses the semantic information contained in the text itself and the keyword information reflecting the text to comprehensively analyses and calculates the similarity between the texts. The experimental results show that the hybrid model can better represent the text information than the single model, and obtain a good F value in the cluster, which effectively improves the text similarity calculation effect.