An event-driven strategy for reactive replica balancing on apache hadoop distributed file system
Rhauani Weber Aita Fazul, Patrícia Pitthan Barcelos · Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing · 2022
Distributed file systems are essential to support applications that handle large volumes of data. One of the most widely used distributed file systems is the HDFS, the Apache Hadoop's Distributed File System. Data replication, which is at the heart of the HDFS storage model, is essential for fault tolerance and performance. As new data is loaded into the system, it is common for the distribution of the replicas among the nodes to become unbalanced. HDFS Balancer is the official solution for data balancing through replica rearrangement. Currently, it is up to the system administrator to monitor the HDFS status and, when considered necessary, run the balancer, which creates a dependency that is inadequate and inefficient in many situations. This work presents an event-based strategy to make the replica balancing process in HDFS transparent. To this end, we have created a metrics observation model and a structure that, based on standardized trigger events, automatically determines when corrective actions should be taken and triggers the reactive balancing process in the file system. The evaluation results demonstrate that the proposed solution is able to keep the cluster balanced, which contributes to overall data reliability and availability. In addition, it allows taking better advantage of data locality during reading operations over the stored data in the HDFS.