Towards quantitative analysis of data intensive computing
Peng Wang, Dan Meng, Zhaoxia Han, Xu Liu · 2011
In modern data centers, Hadoop has been widely used in perform data-intensive computation. Administrators of large scale hadoop clusters leverage statistical data collected at runtime to measure the efficiency of the cluster utilization. In this paper, we propose three statistical metrics - data locality ratio, load balance coefficient and access balance coefficient to quantify performance losses in data intensive applications. We evaluated our metrics using a large scale web click stream application running on a productive hadoop cluster at Tencent Inc.