Bayesian Optimization of Double-SAC Robotic Arm Path Planning Using Recent Prioritized Experience Replay
Shuai Yuan, Ke Wen, Feng Zhang, Jianqiang Xiong · 2023
As for the path planning of robotic arms, Deep Reinforcement Learning (DRL) algorithms have recently been developed. However, the high-dimensional action space and state space will result in reward sparsity and low training efficiency of robotic arm path planning. We propose a Bayesian optimization of Double-SAC algorithm using recent priority experience replay to solve these issues. We first propose the sampling method of recent prioritized experience replay (RPER), which combines recent sampling and prioritized experience replay (PER). The interval of experience pool is divided according to the recent sampling method, and sampling is performed according to the sample priority of the interval. Then we adopt a Bayesian optimization method that reduces the variance of the random strategy and improve the quality of the sampled actions. The larger variance is more beneficial for exploring the environment. Smaller variance is advantageous for exploiting good experiences to obtain higher returns. Our approach finds a good balance between exploration and exploitation. We train two Soft Actor Critic (SAC) agents to obtain two independent probability distributions, and a Bayesian optimizer is adopted to combine the two distributions to design a hybrid strategy. The hybrid strategy can output better actions according to the uncertainty estimation of the distribution, and further enhance training efficiency and stability. Finally, simulation tests are performed on the ROS-Gazebo platform, and the simulation results illustrate that the algorithm proposed in this paper converges faster and has a shorter training time than the original algorithm in terms of path planning performance.