A Novel Interface to a Web Crawler using VB.NET Technology
Iosr Journals, Deepak Kumar1 , Dr. Sushil Kumar2 , Narender Kumar3 · Figshare · 2014
The number of web pages is increasing into millions and trillions around the world. To make searching much easier for users, web search engines came into existence. Web Search engines are used to find specific information on the World Wide Web. Without search engines, it would be almost impossible to locate anything on the Web unless or until a specific URL address is known. This information is provided to search by a web crawler which is a computer program or software. Web crawler is an essential component of search engines, data mining and other Internet applications. Scheduling Web pages to be downloaded is an important aspect of crawling. Previous research on Web crawl focused on optimizing either crawl speed or quality of the Web pages downloaded. While both metrics are important, scheduling using one of them alone is insufficient and can bias or hurt overall crawl process. This paper is all about design a new Web Crawler using VB.NET Technology.