Distributed Online Min-Max Load Balancing with Risk-Averse Assistance
Jingrong Wang, Ben Liang · 2023
Motivated by a wide range of applications from parallel computing to distributed learning, we study distributed online load balancing among multiple workers. We aim to minimize the pointwise maximum over the workers' local cost functions. We propose a novel algorithm termed Distributed Online Load Balancing with rIsk-averse assistancE (DOLBIE), which jointly considers the worker heterogeneity and system dynamics. The workload is distributed to workers in an online manner, where the underloaded workers learn to provide an appropriate amount of assistance to the most overloaded worker for the next online round without making themselves overwhelmed. In DOLBIE, all workers participate in updating the workload simultaneously, and no computationally intensive gradient or projection calculation is required. DOLBIE can be implemented in both the master-worker and fully-distributed architectures. We analyze the worst-case performance of DOLBIE by deriving an upper bound on its dynamic regret. We further demonstrate the application of DOLBIE to online batch-size tuning in distributed machine learning. Our experimental results show that, in comparison with state-of-the-art alternatives, DOLBIE can substantially speed up the training process and reduce the workers' idle time.