Design and Realization of a General Web Crawler
Zhihua Yu · Jisuanji gongcheng · 2005
The paper introduces the design and realization of a general Web crawler, presents the whole structure of the system, analyzes gather, control and data publish, discusses the practical applications. The experiment proves that the system can automate crawl Internet information and provide sufficiency support for following information applications, and have nice generality.