The application of focused crawler based on Heritrix in internet public opinion system
Jian Kun Xiao · Electronic Design Engineering · 2015
In the era of information explosion, the general crawler cannot meet the requirements of personalized search in specific areas, but the topic crawler which can obtain more accurate and comprehensive information get more attention. Among them, the topic crawler as the core part of the vertical search engine has been the research focus in the search direction. On the basis of analyzing the structure and characteristics of the topic crawler Heritrix, this paper it designs a topic crawler by introducing its own improvement suggestions to multithreading and an experiment of the performance of the crawler has been carried out on PC. The results of this experiment proves that the ability, which lays a solid foundation for the development of vertical search engine based on topic crawler.