Research and implementation for topic crawler using statistic model
Ding Yue-wei · Jisuanji gongcheng yu sheji · 2010
Based on the analysis of the existed topic crawlers,another topic crawler using statistic model is proposed,which analyzes the information available and filters the URLs by using the results calculated from the statistic model during crawling,and to effectively addresses the problem how to index the mass web resource and how to find specific topic pages fit user’s interest.The experimental results show that,compared with topic crawlers based on linkage and page content analysis,topic crawler using statistical model can fetch more topic relevant web pages by retrieving less web pages,and improve the crawling accuracy.