Benchmarking Energy and Latency in TinyML: A Novel Method for Resource-Constrained AI
Pietro Bartoli, Christian Veronesi, Andrea Giudici, David Siorpaes, Diana Trojaniello, Franco Zappa · 2025
The rise of IoT has increased the need for on-edge machine learning, with tinyML emerging as a promising solution for resource-constrained devices such as Microcontroller Units (MCUs). However, benchmarking their energy efficiency, latency, and computational capabilities remains challenging due to diverse architectures and application scenarios. Current solutions, such as MLCommons’ “TinyML Perf: Inference” method, have limitations, including the need for separate setups for latency, accuracy, and energy measurements, as well as reliance on energy monitors that power the device, reducing flexibility. Moreover, the absence of a clear distinction between inference and ancillary operations can compromise the accuracy of performance estimations. This work introduces an alternative benchmarking methodology that integrates energy and latency measurements while distinguishing three execution phases—pre-inference, inference, and post-inference—to enable precise profiling. A dual-trigger approach is used to separate these phases, ensuring accurate and repeatable measurements. Additionally, the setup ensures that the device operates without being powered by an external measurement unit, while automated testing can be leveraged to enhance statistical significance. To evaluate our setup, we tested the STM32N6 MCU, which includes a Neural Processing Unit (NPU) for executing convolutional neural networks. Two configurations were considered: one running at maximum performance with the highest clock frequencies and core voltage, and another with reduced settings. The variation of the Energy Delay Product (EDP) was analyzed separately for each phase, providing insights into the impact of hardware configurations on energy efficiency. Each MLPerf model was tested 1000 times to ensure statistically relevant results. Our findings demonstrate that reducing the core voltage and clock frequency improves the efficiency of pre- and post-processing without significantly affecting network execution performance. This approach can also be used for cross-platform comparisons to determine the most efficient inference platform and to quantify how pre- and post-processing overhead varies across different hardware implementations.