Inference-based Decision Making in Games
Tim Rakowski, Marc Toussaint · 2011
Background: Reinforcement learning in complex games has traditionally been the domain of valueor policy iteration algorithms, resulting from their effectiveness in planning in Markov decision processes, before algorithms based on regret minimization guarantees such as upper confidence bounds applied to trees (UCT) and counterfactual regret minimization were developed and proved to be very successful, too. Meanwhile remarkably simple algorithms based on likelihood maximization where found for planning in Markov decision processes, which opened up room for new research. Applying these new methods to extensive games is the focus of this thesis. Results: We describe a generic schema for transforming an extensive game into a multi-agent partially observable Markov decision process (POMDP), derive a strategy update based on the EM algorithm and give an implementation using the hidden Markov model. Tests on a number of minimalistic games suggest that for the two-player case equilibrium strategies are found if the game has pure Nash equilibria but otherwise only the average payoffs of the two players converge to their respective values of a mixed Nash equilibrium, i.e. no equilibrium strategies are found. Further investigation showed that the algorithmic framework is general enough to facilitate the replacement of the M-step by other update procedures such as the polynomial weights algorithm (resulting in external regret minimization) or the counterfactual regret minimization method. Using the latter update, the strategies do converge. Eidesstattliche Erklärung Ich versichere hiermit an Eides Statt, dass diese Arbeit von niemand anderem als meiner Person verfasst worden ist. Alle verwendeten Hilfsmittel wie Berichte, Bücher, Internetseiten oder ähnliches