An HMM-based performance diagnosis approach for Hadoop clusters
Jiacong Li, Ying Wang, Jinke Yu, Shaoyong Guo · 2016
Hadoop has become a popular platform for the management of big data. To provide a healthy Hadoop platform for big data application, an HMM-based approach for performance diagnosis in Hadoop clusters is proposed. We use metrics which are collected under the normal situation to train HMM (Hidden Markov Model), then use this model to detect anomaly based on the probability, which is more accurate than other methods. Through evaluation in a controlled environment running Hadoop clusters, we find our approach can find out the real cause of performance problems in an average 84% precision and 83% recall, which is better than the method based on ARIMA and KNN (k-Nearest Neighbor).