On Exploring the Optimum Configuration of Apache Spark Framework in Heterogeneous Clusters
Ioannis Ballas, Vassilis Tsakanikas, Evaggelos Pefanis, Basil Tampakas · 2021
During the previous decade, both industry and academia have started to apply the Big Data paradigm, exploring the value of data. As the volume of the collected data increases, the required computational infrastructures need to increase their capacities in order to be able to process the data. This work proposes a model for assessing the optimal configuration parameters in a heterogeneous Spark Cluster which is validated against two different use cases. The performed experiments have shown that the proposed model can successfully estimate the optimal Spark configuration parameters, both for memory-intensive and CPU-intensive applications.