Web Crawler: A Review
Md. AbuKausar, Vijaypal Singh Dhaka, Sanjeev Kumar Singh · International Journal of Computer Applications · 2013
Information Retrieval deals with searching and retrieving information within the documents and it also searches the online databases and internet.Web crawler is defined as a program or software which traverses the Web and downloads web documents in a methodical, automated manner.Based on the type of knowledge, web crawler is usually divided in three types of crawling techniques: General Purpose Crawling, Focused crawling and Distributed Crawling.In this paper, the applicability of Web Crawler in the field of web search and a review on Web Crawler to different problem domains in web search is discussed.