An Efficient Block Assignment Policy in Hadoop Distributed File System for Multimedia Data Processing
Cheolgi Kim, Dae-Chul Lee, Jae-Hyun Lee, Jaehwan Lee · IEICE Transactions on Information and Systems · 2019
Hadoop, a distributed processing framework for big-data, is now widely used for multimedia processing. However, when processing video data from a Hadoop distributed file system (HDFS), unnecessary network traffic is generated due to an inefficient HDFS block slice policy for picture frames in video files. We propose a new block replication policy to solve this problem and compare the newly proposed HDFS with the original HDFS via extensive experiments. The proposed HDFS reduces network traffic, and increases locality between processing cores and file locations.