Design and Implementation of University Focused Crawler Based on BP Network Classifier
Hua Jiang, Bing Han, Ying Lin, Dan Zuo, Yong Xing Ge · 2009
The rapid growth of the World-Wide Web poses unprecedented scaling challenges for general-purpose crawlers and search engines. Crawling the Web quickly and entirely is an expensive, unrealistic goal because of the required hardware and network resources. A focused crawler is an agent that targets a particular topic and visits and gathers only a relevant, narrow Web segment while trying not to waste resources on irrelevant material. It can be used to build domain-specific Web search portals and online personalized search tools. In this paper, we describe the design and implementation of a university focused crawler that runs on BP network classifier for prediction of the links leading to relevant pages. We present the flow of the system, discuss the performance, report the experimental results based on it. Our experiments show that the BP classifier performs very well in obtaining accurate relevant university Web resources.