Design of A Multi-strategy Topic Specific Web Crawler
Pan Jin · 2004
The construction, operation and maintenance of Web search engines require lots of resources while they still have disability in providing timely and specific information. The research on topic specific Web hopes to use to reduce the utility of resource as well as to increase the information usage. Related researchers have adopted various methods in designing the specific crawler. In this paper, we discuss our design of specific crawler that adopts multi-strategy. Our crawler is characteristic of low cost of network bandwidth and easy implementation. The experiment reveals that it can synthesize the concern of page relevance and importance and have a good performance in stability. Meanwhile, the crawler is adjustable for choosing preferred strategy, demonstrating good flexibility.