A Novel Snake-like Data Placement Policy for Improving Video Processing in Hadoop Clusters
Eihab SaatiAlsoruji · 2020
Video data include useful information that can be exploited in various video surveillance systems. However, these systems undergo challenges such as needing considerable storage to store a massive volume of video data and significant computing power to process them. Apache Hadoop is a well-known technology that provides distributed storage and parallel processing for large amounts of data, including videos. This paper proposes a novel data placement policy to improve processing video data on Hadoop clusters by minimizing load imbalances among the computing nodes. Change detection is the method used in this paper to process video data by considering the amount of change in different video segments that generate various workloads. The workload diversity leads to load imbalances even if the video segments, having the same size, are equally distributed among the processing nodes of a homogeneous cluster. The conducted empirical study shows the performance improvement achieved by the proposed data placement policy.