Cooperative Model-Based Reinforcement Learning for Approximate Optimal Tracking
Max L. Greene, Zachary I. Bell, Scott Nivison, Jonathan P. How, Warren E. Dixon · 2021
This paper provides an approximate online adaptive solution to the infinite-horizon optimal tracking problem for a set of agents with homogeneous dynamics and common tracking objectives. Model-based reinforcement learning is implemented by simultaneously evaluating the Bellman error (BE) at the state of each agent and on nearby off-trajectory points, as needed, throughout the state space. Each agent will calculate and share their respective on and off-trajectory BE information with a centralized estimator, which computes updates for the approximate solution to the infinite-horizon optimal tracking problem and shares the estimate with the agents. In doing so, the computational burden associated with BE extrapolation is shared between the agents and a centralized updating resource. Edge computing is leveraged to share the computational load between the agents and a centralized resource. Uniformly ultimately bounded tracking of each agent's state to the desired state and convergence of the control policy to the neighborhood of the optimal policy is proven via a Lyapunov-like stability analysis.