Resource Allocation for Batched Multiuser Edge Inference with Early Exiting
Zhiyan Liu, Qiao Lan, Kaibin Huang · 2023
This work considers multiuser edge inference for providing inference services to multiple users at the wireless edge. Multiple tasks are uploaded and grouped into a single batch for parallel processing at the edge server, while a task may exit early from the neural network without traversing the whole model. To efficiently grant users with heterogeneous requirements on accuracy and latency, we study in this paper the joint allocation of communication-and-computation (C2) resources. Two efficient algorithms are designed under the criterion of maximum throughput. First, consider the case with batching but without early exiting. The target problem is optimally solved using a proposed algorithm that nests a threshold-based scheme, which selects users with the best channels and meeting the computation-time constraints, in a sequential search for the maximum batch size. Next, consider the general case with batching and early exiting. A low-complexity sub-optimal algorithm for C2resource allocation is developed by modifying the preceding algorithm to exploit early exiting for latency reduction. Experimental results demonstrate that the proposed C2resource allocation algorithms can leverage batching and early exiting to achieve 1.95x throughput over conventional schemes.