Anti-Crawler strategy and distributed crawler based on Hadoop
HongRu Wang, Chunfang Li, LingFei Zhang, Minyong Shi · 2018
With the exponential growth of network information resource, distributed Web Crawler was introduced for fetching massive web pages. However, when many websites designed a series of Anti-crawlers tactics to disturb Web Crawlers, it is our first imperative to deal with these strategies. We first elaborated on Web-Crawlers and Anti-Crawlers strategies. At the same time, this paper mainly proposed some effective measures to guarantee efficient web-crawling when faced with Anti-crawler tactics. Then we designed a experiment to analyze possible influencing factors. Finally, the future direction about distributed Web Crawels which runs on Hadoop platform was proposed by this paper.