An Improved Crawler Algorithm Based on Hierarchical Structure Preservation
Zhi Feng Hao, Ze Bin Zhang, Zhao quan Cai, Han Pang Huang · Key engineering materials · 2011
This paper proposes an improved web crawler algorithm to climb more useful information since the basic web crawler algorithm is low-efficiency and easy to climb useless repeated information. By the proposed algorithm, the website urls are hierarchical saved to store websites overall topology, which will make crisscross complex web URL system from a graphic structure into a tree structure. The actual website BBS experiments show that the algorithm is much better than the basic web crawler algorithm in crawling speed and download information such as the usefulness of baking. Furthermore, it provides a performing structure mode for the increment crawler algorithm.