Evaluating multi-agent reinforcement learning on heterogeneous platforms
Samuel Wiggins, Yuan Meng, Rajgopal Kannan, Viktor K. Prasanna · 2023
Multi-Agent Reinforcement Learning (MARL) is a key technology in Artificial Intelligence Systems. The development of MARL is a “Training-in-Simulation” process and involves time-consuming computation and communication. Existing implementations of MARL using homogeneous architecture fail to satisfy the above requirements simultaneously. In this work, using a state-of-the-art MARL algorithm - Multi-Agent Deep Deterministic Policy Gradient (MADDPG) - as a benchmark, we provide a systematic analysis of the performance bottleneck on multi-core CPU platforms. We show that (a). the training throughput of large neural network models can benefit from data parallel architecture such as GPU; (b) Memory-intensive components including replay management and all-to-all communication leads to large memory overheads on multi-level cache memory hierarchy on CPU and GPU, thus should be accelerated using custom hardware coupled with large on-chip SRAM with specialized data layout optimizations on FPGA.