Design of Distributed Parallel Web Spider Based on ProActive
Liang Zheng-you · Jisuanji gongcheng · 2008
It becomes more slowly to collect the data by single Web Spider, and higher cost for developing distributed Web Spider by MPI technology or Java technology. This paper designs and realizes a distributed parallel Web Spider with the Active Object, Network parallel computing technology and automatic deployment mechanism provided by ProActive middleware. The experimental results show that the data collection rate of the P-Spider is 2.2 times faster than the data collection rate of the multi-thread Web Spider.