Integrating symmetry of environment by designing special basis functions for value function approximation in reinforcement learning
Guofang Wang, Zhou Fang, Bo Li, Ping Li · 2016
Reinforcement learning (RL) is usually regarded as tabula rasa learning, and the agent needs to randomly explore the environment, so the time consuming and data inefficiency will hinder RL from the real application. In order to accelerate learning speed and improve data efficiency, in this paper we expand the symmetry definition from finite state space to infinite state space and then propose designing a special type of symmetric basis functions for value function approximation to integrate the prior knowledge of symmetry about the environment for large or even infinite state space. After that, as an example, this particular approximate structure is incorporated into the policy evaluation phase of Least-Square Policy Iteration (LSPI), which we call symmetric LSPI (S-LSPI) and the convergence property is analyzed. Simulation results of chain walk and inverted pendulum balancing demonstrate that in contrast with regular LSPI (R-LSPI), the convergence speed of S-LSPI increases greatly and the computational burden decreases significantly simultaneously. It can illustrate the use of symmetric basis functions to capture the property of symmetry very well, and as a case study, it shows the promise to integrate symmetry of environment into RL agent.