Research on a new topic crawler based on HITS algorithm and semantic fusion
Peng Li, Tianling Qiao, Yonxing Guang · 2020
The topic crawler needs to filter for Web prior to crawl Web page, and the crawl strategy of traditional topic crawler is to evaluate the content or evaluate the link relation, for the former topic crawler usually lack of overall importance, it is difficult to reflect the whole situation of the Web, and the theme crawler implemented by the latter will crawl to a large number of topic irrelevant pages in the later stage of crawling. In this paper, a new topic crawler algorithm is designed. The SVSM model is built to calculate the topic relevance of web pages, and HITS algorithm is used to calculate the comprehensive score of web pages. The topic crawler crawls the web page in the order of ranking from highest to lowest. In this paper, the new crawler is compared with two traditional crawlers under the same experimental environment, and the accuracy of the new topic crawler is 23% higher than that of the traditional topic crawler.