Multi-Agent Reinforcement Learning with Asymmetric Representation Assisted by Multi-Objective Evolutionary Algorithms
Ye Tian, Weixin Wang, Shangshang Yang, Panpan Zhang, Xingyi Zhang · 2024
In the face of a series of challenging control tasks, multi-agent reinforcement learning has demonstrated its superiority. However, it suffers from certain drawbacks like deceptive reward functions, unstable training processes, and a lack of exploration for novel policies. To address these issues, the combination of evolutionary algorithms and reinforcement learning has been introduced, as evolutionary algorithms possess good exploration and convergence properties, enabling reinforcement learning to better leverage its performance. To enhance the complementary advantages of evolutionary algorithms and reinforcement learning, we combine them into an asymmetric reinforcement learning framework by utilizing a shared observation encoder, allowing the algorithm's performance to break through certain bottlenecks. In addition to the consideration of reward, we propose the concept of policy novelty, which measures the degree of difference between a policy's behavior and others. Consequently, the training of multiple agents is formulated as a large-scale bi-objective optimization problem, and solved by a multi-objective evolutionary algorithm. The proposed approach is tested on eight experimental tasks within the MA-MuJoCo framework, exhibiting superiority over commonly used approaches.