Basil: A Fast and Byzantine-Resilient Approach for Decentralized Training
Ahmed Roushdy Elkordy, Saurav Prakash, Salman Avestimehr · IEEE Journal on Selected Areas in Communications · 2022
Decentralized (i.e., serverless) training across edge nodes can suffer substantially from potential Byzantine nodes that can degrade the training performance. However, detection and mitigation of Byzantine behaviors in a decentralized learning setting is a daunting task, especially when the data distribution at the users is heterogeneous. As our main contribution, we proposeBasil, a fast and computationally efficient Byzantine-robust algorithm for decentralized training systems, which leverages a novel sequential, memory-assisted and performance-based criteria for training over a logical ring while filtering the Byzantine users. In the IID dataset setting, we provide the theoretical convergence guarantees ofBasil, demonstrating its linear convergence rate. Furthermore, for the IID setting, we experimentally demonstrate thatBasilis robust to various Byzantine attacks, including the strong Hidden attack, while providing up to absolute ~16% higher test accuracy over the state-of-the-art Byzantine-resilient decentralized learning approach. Additionally, we generalizeBasilto the non-IID setting by proposing Anonymous Cyclic Data Sharing (ACDS), a technique that allows each node to anonymously share a random fraction of its local non-sensitive dataset (e.g., landmarks images) with all other nodes. Finally, to reduce the overall latency ofBasilresulting from its sequential implementation over the logical ring, we proposeBasil+that enables Byzantine-robust parallel training across groups of logical rings, and at the same time, it retains the performance gains ofBasildue to sequential training within each group. Furthermore, we experimentally demonstrate the scalability gains ofBasil+through different sets of experiments.