Spark’s operation time predictive in cloud computing environment based on SRC-WSVR
Peng Li, Lu Dong, He Xu, Ting Fung Lau · Journal of High Speed Networks · 2018
In Apache Spark cloud computing environment, computation capability of each node varies, together with the data size and uncertainties in application execution, these result in the differences in task execution time of a job. In order to enhance the accuracy in load execution time prediction, and r easonably guide the user to apply for Spark cluster resources, this paper researches on the execution flow of Spark job, collects the load time consumption index, and puts forward the time index fusion calculation scheme. And then, this paper researches on the multiple linear regression model and support vector machine model to explore the payload execution time and CPU Core, inputting data volume, memory size and other performance indicators. Based on the two models above, this paper proposes a Standard Regression Coefficient-based Weighted Support Vector Regression time prediction model (SRC-WSVR). Finally, through comparing the results from the prediction model proposed in this paper with conventional regression prediction model and Standard Support Vector Machine model, it proves that SRC-WSVR has a higher prediction accuracy, which can provide valid data reference for predicting Spark resource consumption.