Sequential Markov Games With Ordered Agents: A Bellman-Like Approach
Nachuan Yang, Jason J. R. Liu · IEEE Control Systems Letters · 2020
Markov games, as a framework for multi-agent reinforcement learning, has been well investigated in the past decades. In Markov games, all agents simultaneously select actions at each state. However many situations in the real world have to be modeled as a sequential process with ordered agents, which motivates us to widen the view and concepts of Markov games so that all agents can sequentially select actions in some order. This letter studies a step in this direction where exactly two agents with general objectives select actions sequentially in an infinite horizon process. Based on the framework of Markov games, the Bellman-like operators are proposed to analyze the evolutionary process and dynamic programming algorithms are developed to derive the optimal equilibrium for the proposed sequential model. Finally, the effectiveness of the theoretical results is illustrated via numerical simulations.