Budgeted Hierarchical Reinforcement Learning
Aurélia Léon, Ludovic Denoyer · 2018
In hierarchical reinforcement learning, the framework of options models sub-policies over a set of primitive actions. In this paper, we address the problem of discovering and learning options from scratch. Inspired by recent works in cognitive science, our approach is based on a new budgeted learning approach in which options naturally arise as a way to minimize the cognitive effort of the agent. In our case, this effort corresponds to the amount of information acquired by the agent at each time step. We propose the Budgeted Hierarchical Neural Network model (BHNN), a hierarchical recurrent neural network architecture that learns latent options as continuous vectors. With respect to existing approaches, BHNN does not need to explicitly predefine sub-goals nor to a priori define the number of possible options. We evaluate this model on different classical RL problems showing the quality of the resulting learned policy.