Nanily: A QoS-Aware Scheduling for DNN Inference Workload in Clouds
Xuehai Tang, Peng Wang, Qiuyang Liu, Wang Wang, Jizhong Han · 2019
DNN inferences are widely emerging as a service and must run in sub-second latency, which need GPU hardware to achieve parallel accelerating. Prior works to improve the utilization of GPUs in DNN inference services include two methods, resource-centric method, which increases the throughput of a single resource but many requests miss latency SLO under bursty workloads, or request-centric method, in which the resources are allocated dynamically, has the lower throughput on GPU hardware. We present Nanily, a QoS-Aware scheduling that integrates the best of two worlds. Under the particular latency, Nanily adopts adaptive batching to schedule DNN inference requests, and pre-schedule predicted requests to provision the optimal GPU resources, which can not only improve throughput of a single GPU but also achieve GPU resources efficiency. We implement and evaluate our Nanily in simulation experiments that show Nanily reduces the number of GPU resources by more than 45% under meeting latency SLO compared to an aggressive state-of-the-art.