A MapReduce-Enabled Scientific Workflow Framework with Optimization Scheduling Algorithm
Zhuo Tang, Min Liu, Kenli Li, Yuming Xu · 2012
As the data collection volumes growing rapidly, some complex computation are beyond the ability of our classical process methods. A framework combine between MapReduce and workflow can present a good contribution to this problem through parallel processing for the largescale systems. Currently there are several researches on the scheduling policy for this combination framework in homogeneous cluster or simple heterogeneous cluster, however the scheduling on MapReduce-level and workflow-level are detached. Thus we firstly propose a MapReduce-enabled scientific workflow integrated with an optimization scheduling algorithm to consider both level simultaneously and to support complex heterogeneous environment. Our new Model comprise two components: The job prioritizing module to compute the priorities of all jobs, and the task assignment module to allocate suitable slots for each block and schedule the tasks with respect to data-local. We prove by experiment that in our combination framework the new scheduler policy (MRWS) outperforms other polices in this area.