Task Scheduling Algorithm Based on MDP in Big Data Processing Framework
Feng Yanpen · Journal of Shenzhen Polytechnic · 2014
A task scheduling algorithm based on Markov decision process is proposed to address the problem of task scheduling in MapReduce framework.The algorithm describes the load of node in cluster and data localization using state space.The state transfer function represents the influence scheduling strategy of the state.The optimal scheduling policy is obtained by solving the MDP using value iteration.The experimental results show that this algorithm can guarantee the data locality,reduce job response time and improve the overall performance of the system.