Adaptive Importance Sampling with Automatic Model Selection in Value Function Approximation
Hirotaka Hachiya, Takayuki Akiyama, Masashi Sugiyama, Jan Peters · 2007
Off-policy reinforcement learning is aimed at efficiently reusing data samples gathered in the past, which is an essential problem for physically grounded AI as ex-periments are usually prohibitively expensive. A com-mon approach is to use importance sampling techniques for compensating for the bias caused by the difference between data-sampling policies and the target policy. However, existing off-policy methods do not often take the variance of value function estimators explicitly into account and therefore their performance tends to be un-stable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. We demonstrate the usefulness of the proposed approach through simulations.