Self-Configured Workflow Platform for MapReduce Job Execution in Cloud Environment
Muntadher Saadoon, Novia Indriaty Admodisastro · 2022
Hadoop MapReduce becomes the essential framework to store, process and analyse big data in large-scale computing environments. The framework provides numerous opportunities to handle data-intensive applications like web crawling and indexing, data mining, anomaly prediction and machine learning. However, manual configurations of the framework parameters expose to human errors and lead to performance overhead and resource consumption penalties. Existing deployment automation solutions intended to mitigate the limitations by only facilitating the application execution workflow without considering crucial conditions that violates the overall performance after setting up the infrastructure parameters. This paper proposes a self-configured workflow platform with a user-friendly collaborating environment that provides automated setups of the framework paraments to obtain ideal performance compared to the native deployment of Hadoop MapReduce. The self-configured workflow platform has been evaluated in experiments on a real Hadoop cloud-based cluster and the results shows that the execution times are comparable to the native application.