RankMass crawler: a crawler with high personalized pagerank coverage guarantee
Junghoo Cho, Uri Schonfeld · Very Large Data Bases · 2007
Crawling algorithms have been the subject of extensive research and optimizations, but some questions remain open. In particular, given the unbounded number of pages available on the Web, search-engine operators constantly struggle with the following vexing questions: When can I stop downloading the Web? How many pages should I download to cover of the Web? How can I know I am not missing an part when I stop? In this paper we provide an answer to these questions by developing, in the context of a system that is given a set of trusted pages, a family of crawling algorithms that (1) provide a theoretical guarantee on how much of the important part of the Web it will download after crawling a certain number of pages and (2) give a high priority to pages during a crawl, so that the search engine can index the most part of the Web first. We prove the correctness of our algorithms by theoretical analysis and evaluate their performance experimentally based on 141 million URLs obtained from the Web. Our experiments demonstrate that even our simple algorithm is effective in downloading pages early on and provides high coverage of the Web with a relatively small number of pages.