Coordinated Multi-Agent Learning for Decentralized POMDPs
Chongjie Zhang, Victor Lesser · 2012
In many multi-agent applications such as distributed sensor nets, a network of agents act collaboratively under uncertainty and local interactions. Networked Distributed POMDP (ND-POMDP) provides a framework to model such cooperative multi-agent decision making. Existing work on ND-POMDPs has focused on offline techniques that require accurate models, which areusuallycostlytoobtaininpractice.Thispaperpresents a model-free, scalable learning approach that synthesizes multi-agent reinforcement learning (MARL) and distributedconstraintoptimization(DCOP).ByexploitingstructuredinteractioninND-POMDPs,ourapproach distributes the learning of the joint policy and employs DCOP techniques to coordinate distributed learning to ensure the global learning performance. Our approach can learn a globally optimal policy for ND-POMDPs with a property called groupwise observability. Experimental results show that, with communication during learning and execution, our approach significantly outperforms the nearly-optimal non-communication policies computed offline.