Focused Crawling Oriented Multi-Granular Priority Computation for URLs
Jingsheng Lei · Zhongwen xinxi xuebao · 2009
The performance of the focused crawler is crucial to a vertical search engine.Two scientific computation issues to be addressed in the design of focused crawlers are:(1) how to compute the relevance of a current visited Web page to a given topic,(2) how to compute the priorities of unvisited URLs in the queue.For the first issue,this paper describes the calculation of the relevance of a page to the topic based on the page's topical text blocks and related link blocks.For the second one,a novel approach is proposed to prioritize these unvisited URLs by hierarchical topic context of four different granularities,i.e.site level,page level,block level and link level.Finally,a new focused crawling algorithm is presented.Experiments show that the new algorithm is more effective than three traditional algorithms in terms of precision rate and information amount without increasing time complexity.