Design Issues for Search Engines and Web Crawlers: A Review
Deepak Kumar · IOSR Journal of Computer Engineering · 2013
The World Wide Web is a huge source of hyperlinked information contained in hypertext documents.Search engines use web crawlers to collect these web documents from web for storage and indexing.The prompt growth of the World Wide Web has posed incomparable challenges for the designers of search engines and web crawlers; that help users to retrieve web pages in a reasonable amount of time.In this paper, a review on need and working of a search engine, and role of a web crawler is being presented.