Towards Dynamic and Optimal Big Data Placement
Kyriakos Kritikos · 2018
Nowadays, data are being produced at a very fast pace. This leads to the generation of big data that need to be properly managed, especially due to the increased complexity that their size introduces. Such data are usually subject to further processing to obtain added-value knowledge out of them. Current systems seem to focus more on how to more optimally perform this processing while they neglect that data placement can have a tremendous effect on the processing performance. In this respect, big data placement algorithms have been already proposed. However, most of them are either suggested in isolation to the big data processing system or are not dynamic to deal with required big data placement changes at runtime. As such, this paper proposes a novel, dynamic big data placement algorithm which can more optimally find the best placement solution by considering multiple optimisation objectives and solving in a more precise manner the big data placement problem with respect to the state-of-the-art. Further, a novel suggestion for optimally combining such an algorithm with a big data application management system is proposed so as to have the ability to address in conjunction both big data placement, processing and resource management issues. Respective experimental evaluation results showcase the efficiency of our algorithm in producing optimal big data placement solutions.