A Repair Pipeline Method for Erasure Codes Based on Complete Node Load Balancing
Yan Wang, Junhao Shi, Yanhua Cao, Hao Wang · 2024
In distributed storage systems, the repair pipeline has become one of the important data transmission methods in erasure coding repair due to its advantages of improving the computational efficiency and reducing the repair time. In order to solve the problem of load imbalance of helper nodes in the traditional repair pipeline structure, i.e., when some of the nodes are idle, some other nodes are transformed into bottleneck nodes due to heavy workload, which leads to the reduction of repair performance. In this regard, this paper proposed a new repair pipeline structure, CNLB-RP, which adopts the data slicing technique to group and stagger the placement of each transmission task, and cleverly arranges the data transmission process, making it possible to equalize the read and write loads of each helper node under more relaxed conditions (the number of data slices is an integer multiple of the number of helper nodes). Experimental results show that CNLB-RP can achieve complete read and write load balancing among the helper nodes in the pipeline structure. And meanwhile, CNLB-RP can improve data reliability and is more suitable for distributed storage systems.