Parallel DNNs on Hybrid Heterogeneous Platforms: Design, Implementation, and Optimisation for Performance and Energy
Hamidreza Khaleghzadeh, Atefeh Khazaei, Alexey Lastovetsky · IEEE Access · 2026
Deep Neural Network (DNN) packages are designed for homogeneous platforms, limiting their ability to leverage different types of processing resources simultaneously. This article first proposes a hybrid programming model for designing and implementing parallel and portable heterogeneous DNN applications, facilitating DNN parallel training on hybrid platforms. It then explores the performance and energy efficiency of the proposed methodology on hybrid platforms. We implement a fully-connected multilayer perceptron DNN application based on the proposed methodology and study its performance and energy behaviour for different distributions of batches between computing devices. Our findings highlight the outstanding impact of the distribution of batches on the execution time and energy consumption of DNN training, making it a critical decision variable for optimisation. We then mathematically formulate the bi-objective optimisation problem for the proposed DNN methodology, targeting both performance and energy consumption objectives, and obtain Pareto-optimal solutions for these two objectives using two algorithms. The performance improvement and energy saving of the hybrid DNNs are validated on heterogeneous servers. Our results demonstrate that solving the bi-objective optimisation problem yields a wide range of trade-off solutions between execution time and energy consumption, where only one solution achieves load balancing, and others achieve partial load balancing across computing devices. Furthermore, adding additional processing resources enhances DNN training performance, but using less energy-efficient devices, while improving performance, leads to increased energy consumption for performance-optimised solutions. We also compare our approach with TensorFlow’s workload distribution, and the results show that our batch distribution delivers superior performance and energy efficiency.