Starvation Avoidance with Deadline Constraints in Hadoop Environments
Alexandru-Ciprian Farcasanu, Florin Pop, Mihaela-Catalina Nita, Ciprian Dobre · 2017
MapReduce model was designed for distributed large volume of data processing. Time constraints are very important for user productivity but, in a shared cluster, the need for job starvation avoidance also arises. In this paper we propose an extension to the well-known FairScheduler algorithm from Hadoop which takes into consideration soft deadlines for jobs in homogeneous clusters, aiming to improve productivity by better satisfying the user time needs. Our model is as follows: when a job is launched, a deadline is provided. When the job starts running, it has a default priority assigned by which FairScheduler splits resources. The job gets allocated resources and at a certain moment in time it has a current execution speed. Based on speed, it is computed how many more resources the job needs to finish in time. Given this, the job priority is dynamically adjusted so the FairScheduler's resource division policy can meet the deadlines. We validated our algorithm by execution in a real Hadoop environment and we obtained increased performance under deadline constraints while also avoiding job starvation.