An Effective Parallel Web Crawler based on Mobile Agent and Incremental Crawling
Mohammad Abu Kausar, Vijaypal Singh Dhaka, Sanjeev Kumar Singh · Journal of Industrial and Intelligent Information · 2013
A huge amount of new information is placed on the Web every day. Large scale search engines frequently update their index gradually and are not capable to present such information in a timely behavior. An incremental crawler downloads customized contents only from the web for a search engine, thereby helps falling the network load. This network load farther will be reduced by using mobile agents. It is reported in the previous literature that the 40% of the current Internet traffic and bandwidth utilization is due to these crawlers. These crawlers also effect load on the remote server by using its CPU cycles and memory, these loads must be taken into account in order to get high performance at a reasonable cost. This paper deal with all those problems by proposing a system based on parallel web crawler using mobile agent. The proposed approach uses mobile agents to crawl the pages. The main advantages of parallel web crawler based on Mobile Agents are that the analysis part of the crawling process is done locally. This drastically reduces network load and traffic which can improve the performance and efficiency of the crawling process.