Heet: Accelerating Elastic Training in Heterogeneous Deep Learning Clusters
Zizhao Mo, Huanle Xu, Chengzhong Xu · 2024
Modern GPU clusters inherently exhibit heterogeneity, encompassing various aspects such as computation and communication. This heterogeneity poses a significant challenge for the elastic scheduling of deep learning workloads. Unfortunately, existing elastic schedulers often overlook the impact of heterogeneity on scaling efficiency, resulting in considerably prolonged job completion times.