Research on Tunnel Phenomenon in Focused Crawler
Chun Jie Yan, Hang Zhang · 2010
With the rapid development of search engine technology, the technology of focused crawler has been widely concerned and deeply researched. In this research field, there is a "tunnel phenomenon" seriously affecting the crawler's efficiency and coverage. This paper proposes a simple algorithm on correlation calculation and a good method on dynamic adjustment of the threshold, which can effectively solve the problems brought by the tunnel phenomenon. The method works well verified by the experiments.