A SURVEY ON WEB CRAWLER
Jaira Dubey, Divakar Singh · 2013
In today scenario World Wide Web is flooded with huge amount of information. Finding useful information from the Web is quite challenging task. There are many search engines available in the market that will solve our purpose. However among all, selecting proper search engine with highly effective web crawler is quite necessary. There are many challenges in the design of high performances web crawler like it must be able to download pages at high rate, store it into the database efficiently and also crawl page rapidly. In this paper we present taxonomy of WebCrawler, various challenges and solutions of WebCrawler and various crawling algorithms.