NASFLY: On-Device Split Federated Learning with Neural Architecture Search
Chao Huo, Juncheng Jia, Tao Deng, Mianxiong Dong, Zhanwei Yu, Di Yuan · 2024
The integration of Artificial Intelligence (AI) and Internet of Things (IoT) devices has given rise to IoAT, promising transformative applications across various domains. Federated Learning (FL) and Split Learning (SL) are pivotal in harnessing the potential of IoAT, enabling decentralized model training while preserving data privacy. However, the heterogeneity and scalability challenges in IoAT environments necessitate advanced frameworks. In this paper, we introduce an integration of block-wise Neural Architecture Search (NAS) with a multi-partition SFL framework, called NASFLY. This approach uses only devices for actual model training and offers flexible scaling of model fragments to accommodate a wide range of device capabilities. In particular, NASFLY allows devices to utilize idle periods during the lengthy SFL forward and backward propagation phases. This is achieved by employing auxiliary model components dispatched from the server to conduct local supernet elastification using the device’s local dataset. Our method alternates between SFL for backbone network optimization and the local supernet elastification within NAS, where knowledge from the backbone network is transferred to the local supernet branches using distillation techniques. We also propose a device clustering algorithm to further improve training efficiency. Our experimental results demonstrate that this methodology significantly enhances device utilization and improves training efficiency compared with the conventional SFL.