Causality-Guided Exploration for Multi-Agent Reinforcement Learning
Zhonghai Ruan, Chao Yu · 2024
Exploration is crucial for deep reinforcement learning and has garnered significant attention. In multi-agent environments, exploration becomes especially challenging due to the increased complexity of agent interactions and state space. However, existing exploration methods for multi-agent reinforcement learning (MARL) still cannot effectively identify states worth exploring, resulting in a large amount of inefficient exploration. In this work, we propose Causality-Guided multiagent Exploration (CGE), a novel framework that enhances multi-agent exploration by leveraging the causal relation between the agents and the environment. The key insight of CGE is that exploration becomes more effective when agents understand how their actions influence the environment. To this end, we introduce a state-dependent measure of causal influence based on conditional average treatment effect and demonstrate that it reliably guides agents to take impactful actions. Experimental results demonstrate that our approach achieves superior performance compared to state-of-the-art MARL baselines across various tasks in both Google Research Football (GRF) and the StarCraft Multi-Agent Challenge (SMAC).