Explaining Deep Reinforcement Learning Policies with SHAP, Decision Trees, and Prototypes
Kristoffer Eikså, Jan Erlend Vatne, Anastasios M. Lekkas · 2024
The increasing use of complex and uninterpretable Artificial Intelligence (AI) models has led to a growing demand for AI model transparency. In response, the research field of Explainable AI (XAI) is growing, intending to increase trust in black box models by explaining the decisions made. In this paper, we apply the XAI methods SHAP, decision trees, and ProtoDash to Deep Reinforcement Learning (DRL) policies trained in software environments simulating road vehicle traffic cases of different complexity. The SHAP algorithms Deep SHAP and Kernel SHAP allowed us to assess the rationality of the DRL agents' policy by assigning feature attributions to their decisions. Limitations were found in manipulating the input of Deep SHAP, while Kernel SHAP demonstrated more flexibility. Decision trees provided insight into the agents' behavior by splitting and classifying policy decisions, revealing rational tendencies in their overall behavior. ProtoDash was used to select states representative of each action, which highlighted weaknesses in the trained policy. The capabilities these XAI methods demonstrated in terms of insight and transparency can be used to increase trust in black-box decision-making, facilitating for industrial adaptation of AI models.