PPSpider: Towards an Efficient and Robust Topic-Specific Crawler Based on Peer-to-Peer Network
Dongfei Liu, Jia Liu · 2009
To satisfy users in specific area, topic-specific Web crawler is becoming more and more popular in the Web data mining and searching with Internet growing exponentially. Meanwhile, a few peer-to-peer based Web search engines are proposed to cope with problems like single point of failure of current centralized architecture crawlers. Relevance of topic has been deeply studied while selection of starting URLs has no significant progress for a long time. This paper introduces ¿PPSpider¿ to achieve high efficiency of Web crawling for peer-to-peer based topic-specific crawler. Each single peer of PPSpider leverages local browsed Web pages as candidates of starting URLs to mitigate the tunnel problem which troubles current Web crawler. To adapt to dynamic network environment, corresponding crawling algorithm and URL transmitting mechanisms were implemented for PPSpider to help accelerating joint crawling process and reducing network communication flow among peers. Evaluation result shows the benefits of PPSpider and existing relevance algorithms can be easily deployed on PPSpider so as to further improve crawling efficiency.