Using time-correlated noise to encourage exploration and improve autonomous agents performance in Reinforcement Learning
Maria J. P. Peixoto, Akramul Azim · Procedia Computer Science · 2021
Intelligent autonomous agents need to know how to carry out actions based on reasoning, perception, and analysis. Therefore, reinforcement learning algorithms guide the agent to reach this goal by performing steps that guarantee the most significant reward. The problem with this approach is that when the agent finds an optimal action with a considerable premium, it tends to stop exploring the environment to guarantee only that great reward. In this way, the agent stops making a great exploration to find new ways and learn alternatives that could generate a bigger bonus in the face of a change in its context. To alleviate this problem, some techniques, such as Soft Actor-Critic (SAC) and Asynchronous Advantage Actor-Critic (A3C), use entropy regulation to improve policy optimization in reinforcement learning. Thus, the greater the entropy, the lower the probability of the agent’s certainty in a given action. Despite entropy, these algorithms are not immune to local optimal and require additional exploration mechanisms. Therefore, this work proposes an approach that helps the agent’s exploitation policy during reinforcement learning training and improves its performance during the test time. We use the latest state-of-the-art (SOTA) approaches, that is, Asynchronous Advantage Actor-Critic (A3C), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC) in this work. We created a time-correlated noise in the agent’s policy network from the use of a random disturbance generated with the Ornstein-Uhlenbeck process. According to the experiments carried out and the results obtained, we can see that our proposal allowed the agent to explore the environment more during training and improve its performance during the testing time, increasing the reward received in different learning contexts.