A replica storage optimal method for HDFS
Rui Ding, Ziyu Li, Jie Yin · 2020
When the Hadoop distributed file system (HDFS) makes load balancing decisions, the rack-aware strategy of HDFS does not take into account the differences of various node servers comprehensively, which leads to the imbalance of cluster load. According to this defect, we proposed an improved copy placement strategy based on SVM (Support Vector Machine). We took the load of each node server, disk I/O and CPU performance of each node in the cluster into comprehensive consideration, used SVM to classify the nodes according to the characteristic value and finally selected the nodes suitable for placing the copies from the better class of nodes. Experimental results show that our method can effectively improve the load balancing performance of HDFS compared with the original replica storage.