Distributed Parameter Optimization Scheduling Strategy and System Design Based on Mesos
Shuo Li, Zhonghua Lu, Yongze Sun, Sungen Deng, Beifang Niu · 2019
This paper implements a distributed parameter optimization system based on Mesos and studies the scheduling strategy of this system. Using the resource interface of Mesos, the system packages a variety of common parameter optimization algorithms and task scheduling strategy into a framework software that can run on Mesos. Aiming at the two-level scheduling mechanism of Mesos, a dynamic scheduling strategy for distributed parameter optimization system in multi-job environment on hybrid deployment cluster is proposed. This paper designs several experiments, and compares the resource scheduling strategy of the architecture software with the FIFO scheduling strategy in the hybrid deployment scenario. This work reduces the difficulty of optimizing distributed parameters in common scenarios such as deep learning in a cluster environment, and improves resource utilization efficiency in multi-task environment.