Prediction of Cloud Server Job Failures using Machine Learning based KNN Classification and LSTM Modelling Methods
Bhushan Golani, Joydeep Datta, Gurdeep Singh · Zenodo (CERN European Organization for Nuclear Research) · 2021
Cloud computing is the use of a network of remote servers hosted on the internet to store, manage and process data rather than a local server or a personal computer. The cloud computing industry has grown to a great extent in recent times, and it is important to make sure that the delivered service does not deviate from the correct intended service. So, to build a reliable cloud service platform, we need to understand and characterize failures. This work aims to understand the reasons for failure and predict failures that might occur in the future. To accomplish this, we use the Google cluster workload trace. Our analysis reveals that an overwhelming number of resources are used by jobs that eventually fail. For prediction, we use Long Short-Term Memory Network (LSTM) to forecast features on which failure depends. We then predict the termination status using the KNN classification model, which was trained on the existing data.