Enhancing Robustness in Multi-Agent Actor-Critic with Double Actors
Xue Han, Peng Cui, Ya Zhang · 2023
In recent years, multi-agent reinforcement learning (MARL) has shown a strong ability to solve complex system tasks. How to solve the overfitting problem of MARL algorithm and how to improve the robustness of MARL algorithm are important issues that need to be considered when the algorithm is turned to practical application. In this paper, we discuss the application potential of two-actor networks in MARL algorithms. First, we design double actors based on MADDPG algorithm, to improve the robustness of learning strategies, which is called DA-MADDPG algorithm. In the DA-MADDPG, the central-decentralized mismatch caused by two actor networks updating with the same Q value is further discussed. A gradient updating optimization method for DA-MADDPG by KL divergence and clipped function is proposed respectively. Experimental results in MPE environment show that DA-MADDPG and DA-MADDPG bycclipped function can achieve better results than MADDPG algorithm.