Research on Detection Algorithm of WEB Crawler
Hongyan Zhao · International Journal of Security and Its Applications · 2015
In the research of Web crawler, the most important things are structure design and solution of the key technologies.Based on the work of other people, we described the structure design of a distribute Web crawler, which including the organization of hardware and module partition of software.In this paper, one PC is utilized as the main node, and other PCs as the common nodes which are connected in LAN.The software architecture included main node design and common node design.Then, we analyzed solutions of the major techniques of the distributed Web crawler, such as how the nodes of the crawler cooperate with each other, how the task is distributed, how to keep the important Web fresh.We have proposed some practicable arithmetic to solve the problems mentioned above.Besides, we implemented a robust, distensible, customized, distributed Web crawler, and anatomized it.At last, we gave the results of two experiments, including common test and a site download test.