Performance Modeling for Spark Using SVM
Ni Luo, Zhibin Yu, Zhendong Bei, Chengzhong Xu, Chuntao Jiang, Lingfeng Lin · 2016
At present, Spark is widely used in a number of enterprises. Although Spark is much faster than Hadoop for some applications, its configuration parameters can have a great impact on its performance due to the large number of the parameters, interaction between them, and various characteristics of applications as well. Unfortunately, there is not yet any research conducted to predict the performance of Spark based on its configuration sets.In this paper, we employ a machine learning method-Support Vector Machine(SVM) to build performance models for Spark. The input of configuration sets is collected by running Spark application previously with randomly modified and combined parameter values. In this way, we also determine the range of each property and gain a deeper understanding about how these properties work in Spark. We also use Artificial Neural Network to model the performance of Spark and find that the error rate of ANN is on average 1.98 times that of SVM for three workloads from HiBench.