Scaling Disk Failure Prediction via Multi-Source Stream Mining
Shujie Han, Zirui Ou, Qun Huang, Patrick P. C. Lee · 2024
Traditional disk failure prediction approaches struggle to scale with data growth, as they treat data as a whole collection to obtain the global data view for preprocessing and training. Existing distributed machine learning and stream mining systems are designed to scale data processing, particularly for training. However, scaling disk failure prediction faces challenges in the scalability of preprocessing, including additional data movements from data collection to training, data inflation during preprocessing, and multiple-to-multiple data allocation. To address these challenges, we present SCALEDFP, a general framework for scaling disk failure prediction via multi-source stream mining based on three techniques: near-data preprocessing, random downsampling, and training data allocation. SCALEDFP scales disk failure prediction with the number of data sources. It achieves significant throughput gains of preprocessing and training with comparable prediction accuracy against a state-of-the-art disk failure prediction approach that collects data in a centralized place.