Exploiting Best-Match Equations for Efficient Reinforcement Learning
Seijen, H.H., Whiteson, S., Hasselt, Hado, Wiering, M.A. · UvA-DARE (University of Amsterdam) · 2011
This article presents and evaluates best-match learning, a new approach to reinforcement learning that trades off the sample efficiency of model-based methods with the space efficiency of modelfree methods.Best-match learning works by approximating the solution to a set of best-match equations, which combine a sparse model with a model-free Q-value function constructed from samples not used by the model.We prove that, unlike regular sparse model-based methods, bestmatch learning is guaranteed to converge to the optimal Q-values in the tabular case.Empirical results demonstrate that best-match learning can substantially outperform regular sparse modelbased methods, as well as several model-free methods that strive to improve the sample efficiency of temporal-difference methods.In addition, we demonstrate that best-match learning can be successfully combined with function approximation.