Communication-Efficient and Resilient Distributed Deep Reinforcement Learning for Multi-Agent Systems
Jietong Yao, Xin Gong · 2024
In this paper, we address the challenge of developing a resilient Deep Q-Network (DQN) algorithm for multi-agent systems (MASs) under Byzantine attacks. Traditional Q-learning methods suffer from significant communication overhead and limited capacity to handle high-dimensional state spaces, which restricts their scalability and effectiveness in large-scale MASs. To mitigate these issues, we propose an event-triggered resilient DQN (ET-RDQN) algorithm that reduces unnecessary communication while maintaining robustness under the environment exists adversarial agents. Our approach extends the DQN algorithm to MASs and employs the Mean-Subsequence Reduced (MSR) algorithm to filter out extreme values, ensuring that normal agents can still approximately estimate the optimal state-action values and achieve consensus even in the presence of malicious agents. The algorithm incorporates an event-triggered mechanism that triggers communication only when the triggering conditions are met, thereby reducing communication overhead. Theoretical analysis and extensive simulations demonstrate that our ET-RDQN algorithm significantly improves the resilience and efficiency of MASs in the presence of Byzantine attacks, providing a scalable and effective solution for distributed reinforcement learning in adversarial environments. Finally, we conclude this paper and outline potential directions for future work.