A Crawling Algorithm Based on Topical Similarity for Guiding the Web Crawler Though Tunnels
Ya Zhou · Computer Engineering and Science · 2009
Tunneling is always the difficulty of topical web crawling.On the basis of analysing the Web topical features and the shortcomings of the general tunneling technology,this paper raises the algorithm using topical similarity to guide the web crawler though tunnels,and improves the accuracy of topical similarity using the Naive Bayesian classifier.The experimental results show that this algorithm is better than the general tunneling technology in precision and recall rate.