Safety Monitoring of Deep Reinforcement Learning Agents
Amirhossein Zolfagharian, Manel Abdellatif, Lionel Briand, S. Ramesh · 2024
Problem. Deep Reinforcement Learning (DRL) algorithms are increasingly being used in safety-critical systems. Ensuring the safety of DRL agents is a critical concern in such contexts. However, relying solely on testing is not sufficient to ensure safety as it does not offer guarantees. Building safety monitors is one solution to alleviate this challenge. Existing safety monitoring techniques for regular software systems often rely on formal verification to ensure compliance with safety constraints [4]. However, when it comes to DRL policies, formally verifying their behavior to satisfy safety properties becomes an NP-complete problem [6]. Further, monitoring DRL agents in a black-box manner is practically important, as testers and safety engineers often do not have full access to the internals nor the training dataset of the DRL agent [2, 8].