Spatio-Temporal Rule Constraint Guided Safe Reinforcement Learning for CPS
Chan Yin, Yi Zhu, Jinyong Wang, Guosheng Hao · 2023
Cyber Physical System (CPS) integrates computational and physical processes, representing a next-generation intelligent system that combines environmental perception, computational communication, and control decision-making. During the design phase of the system, the formal method ensures that the system satisfies logical specifications and security requirements. In the operational phase of the system, deep reinforcement learning is widely employed for decision-making. However, when faced with uncertain scenarios and complex decision-making tasks, black-box-based deep reinforcement learning systems cannot guarantee safety, and spatio-temporal rule requirements cannot be captured by existing specifications. To overcome these challenges, we propose a spatio-temporal rule constraint guided safe reinforcement learning for CPS. Firstly, we extend the formal language used to describe the system specification to encompass spatio-temporal properties, enabling it to adequately capture the spatio-temporal rule requirements of CPS. Secondly, we introduce the spatio-temporal rule constrained hybrid automaton by extending the spatio-temporal properties onto the hybrid automaton to formalize the modeling of CPS and validate it through the model checking algorithm. Thirdly, on the basis of the validated model, a safe reinforcement learning method combined with formal model for spatio-temporal rule constraint guidance is proposed.