Download performance optimization in Hadoop distributed file system based on P2P
Xingyao Yang · Journal of Computer Applications · 2011
The data block storage mechanism and downloading process in Hadoop Distributed File System(HDFS) cluster were analyzed.In combination with multi-point and multi-threaded Peer-to-Peer(P2P) download idea,an efficiency optimization algorithm was proposed from the aspects of data-block,file and cluster.Concerning the possible imbalanced load problem caused by multi-thread download in HDFS cluster,a download-point selection algorithm was put forward to optimize the download-point selection.The mathematical analysis and experiments prove that the three methods can improve the download efficiency and download-point selection algorithm can achieve loading balance among DataNodes in HDFS cluster.