A Novel Architecture of a Parallel Web Crawler
Shruti Sharma, Ashok Kumar Sharma, Juhi Gupta · International Journal of Computer Applications · 2011
Due to the explosion in the size of the WWW [1,4,5] it becomes essential to make the crawling process parallel.In this paper we present an architecture for a parallel crawler that consists of multiple crawling processes called as C-procs which can run on network of workstations.The proposed crawler is scalable, is resilient against system crashes and other event.The aim of this architecture is to efficiently and effectively crawl the current set of publically indexable web pages so that we can maximize the download rate while minimizing the overhead from parallelization