The Dynamically Efficient Mechanism of HDFS Data Prefetching
Shaochun Wu, Guobing Zou, Honghao Zhu, Xiang Shuai, Chen Liang, Bofeng Zhang · 2013
In recent years, along with cloud computing developing as a widely used computing paradigm, Hadoop Distributed File System (HDFS) has become one of the mandatory techniques, which has many important features, such as master and slave construction of HDFS, direct client accessing, and multi-duplicate of each data block. All of these make HDFS data prefetching much harder than the traditional data acquisition approaches. Moreover, the basic problems of HDFS data prefetching mainly include what kind of data to prefetch, where to prefetch data, how many data to prefetch, and the balance of prefetching data services and normal data access conflicts. Under above analysis, this paper tries to solve these problems and propose the mechanism of the two-layer HDFS data prefetching. The experimental results show that the Hadoop platform which offers data prefetching mechanism can improve 60% of whole performance on data prefetching.