Consistent Hashing Algorithm Based on Slice in Improving Scrapy-Redis Distributed Crawler Efficiency
Zhe-Yu Ying, Fengli Zhang, Qingyu Fan · 2018
Scrapy-Redis is widely used in internet data acquisition. A great distributed crawling framework is embodied in its timeliness, security and reliability. However, the current static Scrapy-Redis crawling framework is not well adapted to changing situations, thus having a poor performance. Therefore, this paper proposes an improvement, first by logically dividing each Redis storage node into a group, adopting a master-slave mode within the group, then combined with dynamic load balancing technology, designed a consistent hashing dynamic load balancing strategy based on data slice, where a slice can dynamically do split and merge operation for accommodating dynamic situations. Experiments show that the strategy retains the advantages of consistent hashing, at the same time optimizes the overall load balancing effect of consistent hashing strategies, and also improves the social network data crawling timeliness and performance of distributed crawlers.