Dynamic Model Reduction for Decentralized Learning in Heterogeneous Mobile Networks
Sangsu Lee, Xi Zheng, Jie Hua, Haoxiang Yu, Christine Julien · 2023
Collaborating with nearby devices to train and personalize deep learning models opens the potential to support new mobile application scenarios. In emerging decentralized learning algorithms, devices communicate over a peer-to-peer network to share knowledge obtained from local data. However, communication bandwidth, computing power, and the duration for which these connections are available are limited and heterogeneous. In this paper, we explore the feasibility and efficacy of adaptive model reduction strategies for decentralized learning algorithms (Dynamic Reduction (DR)). For this study, we use as an exemplar an existing opportunistic learning algorithm (OppCL) that relies on device-to-device model exchanges to iteratively train a local model based on encounters. In layering model reduction on OppCL, when a device encounters a potential learning partner, it dynamically constructs a model reduction suitable for given computation and communication budget by quantizing weights and building a dropout version of a neural network. We term our new approach DR-OppCL and show that DR-OppCL leads to faster convergence with minimal effort in tuning hyperparameters related to model reduction, using both simulated and real-world mobility traces. While we demonstrate our DR approaches in the context of OppCL, they are generic and can be easily applied to other decentralized learning algorithms.