Dynamic Refresh Strategy for Crawler in Search Engine
Kai Gao · Dianzi xuebao · 2007
As for a search engine,keeping up with the evolving Web is necessary.We concern about modeling on an effective Web page collecting policy and propose an adaptive refresh strategy based on the relevance,which is used to adjust the process.On one hand,we think the refresh behavior follows the properties of the Poisson process and analyze the strategy on how to crawl the Web effectively.Further,the relevance is on the basis of the affiliation detecting and the contents analysis.It is used to adjust the process.This makes the process more targeted.The experimental results validate the feasibility of the approach.