A classification framework for straggler mitigation and management in a heterogeneous Hadoop cluster: A state-of-art survey
Kamalakant Laxman Bawankule, Rupesh Kumar Dewang, Anil Kumar Singh · Journal of King Saud University - Computer and Information Sciences · 2022
Hadoop is the most economical and cheap software framework that allows distributed storage and parallel processing of more extensive data sets. Hadoop distributed file system (HDFS) allows distributed storage and parallel processing of vast data sets using MapReduce. However, Hadoop’s current implementation believes that computing nodes connected in a cluster are homogeneous and distribute the tasks equally. This equal load distribution creates the load imbalance during storage, resource contention during task scheduling, hardware degradation by its excess use, and software misconfiguration during cluster management which are the leading causes of stragglers in heterogeneous Hadoop clusters. Due to hardware heterogeneity, Hadoop’s performance degrades in the heterogeneous environment. In our study, the paper reviews and analyzes significant studies. It presents the new classification taxonomy to broadly classify the existing straggler management and mitigation techniques into two approaches: proactive and reactive. It analyses and compares the state of art studies and identifies their limitation based on their results. Finally, the systematic review discusses the open issues and the potential directions for future work to manage and mitigate stragglers from the heterogeneous Hadoop clusters.