Quicker Q-Learning in Multi-Agent Systems
Adrian Agogino, Kagan Tumer · 2005
Multi-agent learning in Markov Decisions Problems is challenging because of the presence of two credit assignment problems: 1) How to credit an action taken at time step t for rewards received at t ′> t; and 2) How to credit an action taken by agent i considering the system reward is a function of the actions of all the agents. The first credit assignment problem is typically addressed with temporal difference methods such as Q-learning or TD(λ). The second credit assignment problem is typically addressed either by hand-crafting reward functions that assign proper credit to an agent, or by making certain independence assumptions about an agent’s state-space and reward function. To address both credit assignment problems simultaneously, we propose the “Q Updates with Immediate Counterfactual Rewards-learning ” (QUICRlearning) designed to improve both the convergence properties and performance of Q-learning in large multi-agent problems. Instead of assuming that an agent’s value function can be made independent of other agents, this method suppresses the impact of other agents using counterfactual rewards. Results on multi-agent grid-world problems over multiple topologies show that QUICR-learning can achieve up to thirty fold improvements in performance over both conventional and local Q-learning in the largest tested systems. 1