AHA: Adaptive Hadoop in Ad-hoc Cloud Environments
Ryan Liu, Shizhe Lin, Ladan Tahvildari · 2021
Cloud computing has become a highly efficacious and popular method when dealing with large computational demands. To better utilize cloud computing resources, an interesting proposal involves using the available machines for distributed computing in an ad-hoc manner during regular and off-peak hours. Our proposed framework, named AHA, implements resource availability considering (RAC) task speculation within the latest version of Apache Hadoop (3.3.0). The resource availability history of each worker node is stored locally and considered during scheduling of MapReduce (MR) workloads. In addition, a fuzzy-rule based self-tuning solution is also proposed to alleviate the need for manual tuning regarding resource availability consideration. Our preliminary evaluations indicate that AHA is able to decrease execution time by up to 20.2 % for certain MR workloads in ad-hoc cloud settings. Overall, the approach shows potential in addressing this real-world issue as our results are also on average upper-bounded by Apache Hadoop with respect to workload execution time in a simulated ad-hoc cloud environment.