Review on improving performance of Web Crawler and Search System Architecture
Dinesh Rai, Naresh Kumar, Jyoti Mor · SSRN Electronic Journal · 2018
The W3 is a popular medium through which different types of users are disseminating information. The size of web is increasing exponentially today. Therefore, efficient searching of required content in minimal possible time becomes a matter of concern; with the increase in number of users per day. The current Web Crawler used for information retrieval have challenges associated with them. As these crawlers while accessing Web Pages consume network bandwidth, CPU Utilization and, memory utilization and other resources. So to improve the performance of Web Crawlers with reduced usage of resources is the need of the hour. Such Crawlers, having high Precision and Recall and less consumption of available resources are required to be developed.