Direct Gradient-Based Reinforcement Learning: II. Gradient Ascent Algorithms and Experiments
Jonathan Baxter, Lex Weaver, Peter L. Bartlett · 1999
In [2] we introduced GPOMDP, an algorithm for computing arbitrarily accurate approximations to the performance gradient of parameterized partially observable Markov decision processes (POMDPs). The algorithm's chief advantages are that it requires only a single sample path of the underlying Markov chain, it uses only one free parameter 2 [0; 1) which has a natural interpretation in terms of bias-variance trade-off, and it requires no knowledge of the underlying state. In addition, the algorithm can be applied to infinite state, control and observation spaces. In this paper we present CONJPOMDP, a conjugate-gradient ascent algorithm that uses GPOMDP as a subroutine to estimate the gradient direction. CONJPOMDP uses a novel line-search routine that relies solely on gradient estimates and hence is robust to noise in the performance estimates. OLPOMDP, an on-line gradient ascent algorithm based on GPOMDP is also presented. The chief theoretical advantage of this gradient bas...