Research on the Search Strategy of the Web Spider in Topic-Oriented Search Engines
Liang Han · Computer Engineering and Science · 2008
Based on our in-depth research on the search strategy in topic-driven engines and topic dependency judgement algorithms, this article presents a design model of the topic-oriented web spider and analyzes its structure in detail.As the key component of the search strategy for the topic-oriented web spider,the topic dependency judgement algorithms ensure a focused web crawling process of the spider.In the process of the dependency judgement between URLs and topics, a novel URL pruning algorithm called EPR is presented based on an analysis of the anchor text and the related properties.