Design of Crawler's Algorithm and Implementation of Crawler's Program
Liang Song · Jisuanji gongcheng · 2004
The research of crawler's algorithm is a hotspot in search engine. This paper,first analyses the current method of designing crawler's algorithm and realizing crawler's program and concludes its disadvantage and advantage. Then it gives a crawler's algorithm of retrieval web page suitable for medium and small-sized web site and realizes this algorithm by jbuider8.0?It is proved that the program speed of downloading web pages is 188~242/minus and 41.92~74.59KB/second.