On Distributed Web Crawler: Architecture, Algorithms and Strategy
Ling Zhang · Dianzi xuebao · 2002
We describe a large - scale distributed Web Crawler system, i.e. Igloo VI.2. Igloo' s distributed architecture is based on our two-tiered Hash mapping algorithm, so that it can do efficient task partition while at the same time providing dynamic scalability. As the quality of crawled Web pages is an important factor for evaluating crawlers, it employs PageRank value as the evaluation metric of pages to improve its crawling efficiency. This paper also provides a detailed discussion of the performance bottlenecks in crawler systems,and proposes a new URL repository access method based on delayed merging' strategy to enable high-speed crawling. The experiments show Igloo can quickly crawl high-quality Web pages as well as present high performance.