Hadoop Local Tasks Scheduling Optimization Algorithm Based on Logistic Regression Model
Proceedings of 2016 the 6th International Workshop on Computer Science and Engineering · 2016
For a TaskTracker has multiple local tasks available, by default, the scheduler executes those tasks in succession with the order of the tasks to be found, this is inefficient.In order to optimize the local tasks scheduling, this paper presented Hadoop local tasks scheduling optimization algorithm based on Logistic regression model.First, related feature vectors of the local tasks were selected and defined, then, based on the way of machine learning with Logistic regression model, trained these vector to get the weight of each vector to decide the task priority, and updated the model constantly by the overload rules.The experimental results show that the proposed algorithm improves map task data locality, at the same time of reducing job running time.