IOWA: An I/O-Aware Adaptive Sampling Framework for Deep Learning
Shuang Hu, Weijian Chen, Yanlong Yin, Shuibing He · 2024
Training deep DNN models is time-consuming, especially when using large datasets. In the standard model training process, data instances are sampled uniformly and fed into the neural networks. However, not all instances contribute equally to the resulting model, and even the same data instance may affect the model differently in different training iterations. In addition to computational costs, I/O overhead can significantly impact the training speed, particularly for I/O-intensive processes. Given these observations, we propose an I/O-aware sampling metric in this paper. Building on this, we introduce an I/O-Aware Adaptive Sampling Framework (IOWA), which includes data profiling, adaptive data sampling, and redundant data instance replacement to accelerate the training process. Extensive exper-iments demonstrate that, compared to traditional DNN training processes, our approach can achieve up to a 3 x speedup without compromising the resulting model.