Harnessing Temporal Information for Efficient Edge AI
Max Sponner, Lorenzo Servadei, Bernd Waschneck, Robert Wille, Akash S. Kumar · 2024
Deep Learning is becoming increasingly relevant in edge and Internet-of-Things applications. However, deploying models on embedded devices is challenging due to their limited resources, requiring trade-offs between accuracy, latency and power consumption. Early Exit Neural Networks address this by dynamically adjusting model depth during inference, optimizing for each input.However, this requires an at-runtime decision mechanism to select the optimal configuration. Unfortunately, current methods do not fully exploit all available information for decision-making. We introduce the Difference Detection and Temporal Patience mechanisms, which leverage temporal correlations in sensor data to improve inference efficiency. We extend the approach to directly operate on intermediate results, making it applicable to traditional neural networks, further reducing inference and implementation costs.Evaluated on representative edge intelligence use-cases, our methods achieved up to a 32% reduction in latency on ECG data, with an accuracy loss of only 0.13 percentage points. When applied to a speech command detection task, latency reduction reached 44.3%, accompanied by a 1.34 percentage point decrease in accuracy.