Design and Realization of Peer-to-Peer Web Spider
Hongxia Wang · Computer Technology and Development · 2010
With the rapid growth of Internet's massive information,it becomes very significant to enhance the data acquisition and update speed of search engines' information collector:Web Spider.Bottleneck caused by center node is apt to occur in the distributed parallel Web Spider with center node.The peer-to-peer distributed parallel Web Spider named P-Spider2.0 is designed and realized with the active object,network parallel computing technology and automatic deployment of ProActive which is a grid and network parallel distributed computing middleware.Experiments were conducted and the experimental results show that the peer-to-peer Web Spider has higher data collection rate than multi-threaded Web Spider,and is easier to manage and deploy.