Sorting Performance Analysis and Optimization in Map Phase of Hadoop System
Yan Zhou · Computer Knowledge and Technology · 2014
MapReduce framework has been widely used in large-scale data analysis applications. The system is well recognized for its elastic scalability and fine-grained fault tolerance, but its performance not satisfactory. MapReduce can achieve better performance with the allocation of more compute nodes from the cloud to speed up computation. However, this approach is not cost-effective. Users desire a more effective MapReduce framework with both elastic scalability and fault-tolerance.In this paper, we analyze the sorting Performance in the Map Phase of Hadoop System. We proposed and implemented a method to optimize the sorting performance dynamically. The experiments show that the method can improve the performance of MapReduce.