Study on re-implement mechanism based on node-ability in Hadoop
Yaohui Wang, Tao Yang · 2011
Face to massive web log data, single log analysis has been unable to meet people's pursuit of high efficiency, the traditional log analysis based on Hadoop has some drawbacks in resource utilization, etc. Therefore, based on traditional Hadoop task scheduler, a new tasks pre-scheduling mechanism based on node-ability is proposed, the size of the task block is determined according to the ability of each processing node. Thus, the conflicts between job execution time and system resource utilization is balanced, which can protect the jobs of the response time and effective enforcement mechanism to reduce the weight of the additional overhead and improve overall resource utilization.