Research and Implementation of Topic Crawler Based on Hadoop
Rui Gu, Jian Feng Jiang · Applied Mechanics and Materials · 2014
This article proposes distributed topic-focused crawlers on the basis of HDFS and MapReduce, which are the two core technologies of Hadoop. The crawler has the advantages of distributed processing ability, scalability, high reliability. This article analyses the dependency of subject by using the method of conceptual analysis and uses MapReduce to realize webpage crawling and updating webpage. At last using experiments to verify the performance, expansibility and reliability of the system