Noisy-MAPPO: Noisy Credit Assignment for Cooperative Multi-agent Actor-Critic methods
Siyue Hu, Jian Hu, Shih-Wei Liao · 2021
Multi-Agent Reinforcement Learning (MARL) has seen revolutionary breakthroughs with its successful application to multi-agent cooperative tasks such as robot swarms control, autonomous vehicle coordination, and computer games. Recent works have applied the Proximal Policy Optimization (PPO) to the multi-agent tasks, called Multi-agent PPO (MAPPO). However, the MAPPO in current works lacks a theory to guarantee its convergence; and requires artificial agent-specific features, called MAPPO-agent-specific (MAPPO-AS). In addition, the performance of MAPPO-AS is still lower than the finetuned QMIX on the popular benchmark environment StarCraft Multi-agent Challenge (SMAC). In this paper, we firstly theoretically generalize PPO to MAPPO by a approximate lower bound of Trust Region Policy Optimization (TRPO), which guarantees its convergence. Secondly, since the centralized advantage value function in vanilla MAPPO may mislead the learning of some agents, which are not related to these advantage values, called \textit{The Policies Overfitting in Multi-agent Cooperation(POMAC)} problem. We propose the noisy credit assignment methods (Noisy-MAPPO and Advantage-Noisy-MAPPO) to solve it. The experimental results show that the average performance of Noisy-MAPPO is better than that of finetuned QMIX; Noisy-MAPPO is the first algorithm that achieves more than 90\% winning rates in all SMAC scenarios. We open-source the code at \url{this https URL}.