Improving value factorization for multi-agent deep reinforcement learning via individual contribution
Xiong Liqin, Lei Cao, Xiliang Chen, Jun Lai, Xijian Luo, Xiaoyan Wang, Legui Zhang, Haoyang Dong · The Computer Journal · 2025
Abstract Multi-agent credit assignment is a research hotspot in the field of cooperative multi-agent reinforcement learning, and its key is how to accurately measure the individual contribution of each agent in the system to promote multi-agent cooperation. Existing solutions mainly use value function factorization or intrinsic reward mechanism, each of which has its own limitations, and both of them utilize global state information, which is not consistent with the information conditions in the actual confrontation. Therefore, this paper proposes a novel value factorization method for multi-agent deep reinforcement learning, which can solve the problem of credit assignment without using global state information. Our method establishes an explicit individual contribution evaluation mechanism for each agent, which portrays the role of each agent in the system by comparing the differences of joint value functions under different information conditions, so that more important agents get more attention, so as to improve the cooperative ability of agents. Experimental results show that our method outperforms all baselines in terms of learning efficiency and stability in multiple scenarios of StarCraft II, and its performance is comparable to that of the method based on global state information in easy scenarios.