Optimizing Federated Learning Efficiency: Exploring Learning Rate Scheduling in Sparsified Ternary Compression
C Nithyaniranjana Murthy, S H Manjula · 2024
Federated Learning facilitates the collaborative training of a deep learning model by leveraging the combined data of multiple entities, with a focus on maintaining the privacy of individual datasets. The Sparse Ternary Compression (STC) method improves gradient sparsification by integrating downstream compression, ternarization, and optimal Golomb encoding of weight updates. Although STC enhances compression techniques in federated learning, the broader scope of Federated Learning lies within the domain of machine learning. Researchers investigating improvements in STC compression techniques must consider neural network concepts and the impact of hyperparameters, such as learning rate, on convergence speed and accuracy. Increased communication across multiple devices poses challenges in achieving desired accuracy levels. Our research underscores the importance of selecting appropriate learning rate schedules when implementing algorithms like STC. Incorrect choice of learning rate schedules can hinder convergence, even if the algorithm functions effectively, emphasizing the crucial role of hyperparameters in reducing communication costs and achieving desired accuracy levels