Focused crawler based on concept context graph
Xiaolei Li, Yajun Du, Xiaoping Huang, Yufeng Hai · 2023
In order to improve the performance of focused crawler, a topic crawling strategy based on Concept Context Graph(CCG) is proposed. User knowledge background is constructed from the initial topic text returned by zhishi.me Knowledge Graph, and it is transformed into a CCG to guide the crawling of focused crawler. During the crawling process, the crawler treats the hyperlinks it encounters as a Virtual Concept(VC), and then calculates the Sememe Similarity(SS) between the VC and the core concept in CCG based on HowNet. According to the relationship between the SS value and the given threshold, it is decided whether to keep the link or not, and the CCG is updated for the reserved VC. The corresponding hyperlinks of the preserved VC are then stored in the queue to be crawled according to the order of SS value from high to low. Through experiments, it can be seen that the topic crawling strategy can improve the performance of focused crawler.