Measuring Data Transmissions from the Edge for Distributed Inferencing with gRPC
Scott Brown, David Harman, Cleon Anderson, Matthew B. Dwyer · 2023
Over the past several years, there has been a large increase to the number of edge sensing devices gathering various types of data, from images to audio. One of the issues however is getting this data to a device that can quickly perform analysis. Limited network and compute are just a few of the issues that will be taken into consideration, when it comes to offloading data for inferencing. Existing solutions to the limits of edge computation includes, local computation w/ compression and pruned models, cloud offloading, and several distributed methods, including data distributed and model distributed inferencing. Previous work has explored the performance of these options using standard HTTP communication between the edge and external devices, now new network communication protocols are explored specifically gRPC. The impact of utilizing a more lightweight communication protocol to efficiently transfer data from the edge to other devices is explored while specifically focusing on potential drawbacks that may occur. gRPC while shown to perform more efficiently than standard HTTP, however does come at the cost of increased CPU utilization which may impact performance at the edge in select cases. Experiments will be conducted using a testbed which was created to allow definition of both edge, mid-range, and cloud computation devices to be defined each with corresponding compute and memory allocations. Additional, control of the network will be managed via the software defined network allowing for multiple bandwidth cases to be tested for performance comparison in various network constrained environments. Results highlight the performance benefits gRPC in each of the cases showing inferencing speed improvements of the previous HTTP implementation. However several of the cases show better improvements than expected.