Hierarchical Architecture for Multi-Agent Reinforcement Learning in Intelligent Game
Bin Li · 2022 International Joint Conference on Neural Networks (IJCNN) · 2022
The intelligent game has made great achievements while also posing grand challenges to reinforcement learning such as multi-agent coordination, long time horizons, complex action control, sparse rewards, etc. In this paper, we propose a hierarchical architecture learning paradigm that methodologically combines the multi-agent algorithm and single-agent algorithm in multi-agent environments. With learning hierarchical policy and the independence of each level in the model, macro-operation is introduced to reduce the original action space, while skillfully mitigating the scalability issue. Besides, the PER-QMIX algorithm, based on the QMIX, uses the prioritized experience replay mechanism to sample important memories more frequently and thus learn more efficiently. We implement the hierarchical structure with the PER-QMIX algorithm and macro-operation in the complex wargame with extremely sparse rewards. We experimentally validate the effectiveness of the proposed methods in intelligent wargame by demonstrating that our approaches can significantly facilitate and accelerate learning compared with baselines.