Research on focused crawler based on Bayes classifier
Zou Yong-bin · Jisuanji yingyong yanjiu · 2009
Focused crawler is the core technology to implement a focused search engine.This paper made a research on focused crawler based on Bayes classifier,introduced the system architecture and key implementation of a focused crawler based on Bayes classifier.The key parts of the crawler include URL queue,crawling history,page downloader and page analyzer.This paper focused on the page ranking algorithm based on Bayes classifier.Used an improved TF-IDF algorithm to extract the characteristics of the page content,and adopted Bayes classifier to compute the page rank.The experiment results show that Bayes classifier is a better choice than PageRank algorithm to implement a focused crawler,under the condition of large network of resources.