Enhancing Negotiation Policies via Spatio-Temporal Directed Graphs for Autonomous Interaction
Fan Guo, Chuyu Ma, Dezong Zhao, Guochen Liu, Kang Song · IEEE Transactions on Automation Science and Engineering · 2025
Mixed traffic environments, comprising autonomous vehicles (AVs) and human-driven vehicles (HDVs), present substantial challenges for developing negotiation policies. These policies are essential for enabling AVs to make adaptive decisions and achieve harmonious interactions with HDVs in dynamic and complex scenarios. Despite their potential in autonomous driving, reinforcement learning-based decision-making schemes remain insufficient in addressing interaction-critical situations. To overcome this limitation, this study proposes a graph-enhanced negotiation-aware policy optimization (GNPO) framework, which embeds interaction awareness into all key components, including representation, understanding, and response. To support interaction representation, the spatio-temporal directed graph (STDG) captures the dynamic and asymmetric nature of interactions. It integrates multiple directed graph topologies, each encoding distinct social priors, to collectively depict realistic interactive behaviors. Additionally, a multimodal feature extractor fuses environmental perception, interaction cues, and task-related information, enabling comprehensive state understanding. Building on this representation, GNPO incorporates an interaction-critical reward function and an entropy-regularized adaptive policy optimization scheme, both designed to promote harmonious and context-aware behaviors in dense traffic scenarios. The proposed framework is validated in both interaction-intensive simulated environments and real-data-based digital twin scenarios, with its effectiveness quantitatively demonstrated by substantial improvements in success rate, safety, and efficiency. Qualitative evaluations further show that GNPO is capable of generating human-like cooperative and competitive behaviors.