Poster: Adaptive In-Network Inference using Early-Exits
Heewon Kim, Seongyeon Yoon, Sangheon Pack · 2023
In-network (or on-path) inference over programmable data planes allows fast and low-overhead inference in deep neural networks. In this work, we propose an adaptive approach to strike the balance between accuracy and processing cost. To be specific, the confidence score is evaluated at the end of each layer, and an early exit is triggered if the confidence score is sufficiently high. We implement this early-exit scheme over BMv2 software switches and the results demonstrate that the proposed scheme successfully controls the trade-off by making use of the confidence score.