Adaptive focused crawling method based on information gain
Yufang Zhang · Jisuanji yingyong yanjiu · 2012
In combination with information gain,this paper proposed a new adaptive focused crawling method.It set up topic vector T by category tree and topic descriptive article of Wikipedia,and automatically learned and fed back to modify weight of each concept in the topic vector space during crawling,improving topic description.Experimental results show that the method contributes to the focused crawler an incremental crawling ability,it is superior to the adaptive method based on the interest ratio significantly in terms of sum of information,and Web pages crawled with the former are more related to the topic than the latter.