Multi-Agent Reinforcement Learning with Clustering and Experience Sharing
Kaname Inokuchi, Toshiharu Sugawara · Procedia Computer Science · 2024
We propose a training method for a heterogeneous multi-agent system to improve the learning efficiency in sparse-reward environments. Although extensive research on multi-agent deep reinforcement learning are conducted actively, these studies often assume that all agents are homogeneous to share/utilize learning parameters in their networks. Unfortunately, this is not always the case in real-world applications where heterogeneous autonomous agents, i.e., those with different capabilities and perspectives, must properly cooperate and coordinate with each other. In our learning method, which is an extension of the shared experience actor-critic (SEAC) for a heterogeneous agent environment, agents are classified depending on their features (such as trajectories of the observations, actions and received rewards) using variational autoencoder, and share their experience among agents within each cluster to train their individual agents for improving the learning efficiency in a sparse-reward environment. Our experimental evaluation shows that the proposed method is capable of more efficient cooperative/coordinated behaviors than the baselines while remaining the advantages of SEAC.