Analyzing the Interplay Between Random Shuffling and Storage Devices for Efficient Machine Learning
Zhi-Lin Ke, Hsiang-Yun Cheng, Chia-Lin Yang, Han-Wei Huang · 2021
Machine learning algorithms, such as Support Vector Machine (SVM) and Deep Neural Network (DNN), have gained a lot of interest recently. When training a machine learning algorithm, randomly shuffling all the training data can improve the testing accuracy and boost the convergence rate. Nevertheless, realizing training data random shuffling in a real system is not straightforward due to the slow random accesses in hard disk drives (HDDs). Common random shuffling implementations assume that HDD is used as storage, so they sacrifice the random degree of shuffling to reduce random storage accesses. Different from conventional HDD, emerging solid-state drive (SSD) based storage devices, such as Intel Optane SSD, offer fast random accesses. In this paper, we explore the opportunities to take advantage of the fast random access property in SSD to perform full-range random shuffling without taking up precious CPU memory and study the interplay between different shuffling methods and various types of storage devices. We use a lightweight implementation of random shuffling (LIRS) as an example of the SSD-aware shuffling method to conduct performance analysis. Evaluations show that, compared to conventional shuffling methods, LIRS can improve convergence rate and reduce the total training time of SVM and DNN by 67.1% and 33.9% on average.