Reproducible and efficient deep reinforcement learning
Shengyi Huang, Santiago Ontañón · 2023
Deep reinforcement learning (DRL), a paradigm by which agents learn how to do tasks through trial and error, has achieved great success in many domains. Researchers have successfully applied DRL to train autonomous agents that learn to play video games from pixels and control simulated robots, all the way up to design microchips. Despite these impressive accomplishments, DRL algorithms can be hard to reproduce due to their sensitivity to hyperparameters and seemingly unimportant implementation details. Additionally, running DRL algorithms can be computationally inecient, especially in challenging domains such as real-time strategy (RTS) games which pose a significant challenge to DRL due to their large action space, sparse rewards, and partial observability. This thesis makes contributions toward making DRL more reproducible and ecient. First, we study how implementation details of DRL, often left out of academic publications, have a significant impact on algorithm behavior. We then propose a new framework to mitigate reproducibility issues, and this framework is encapsulated in a DRL library called CleanRL. We further identify and address reproducibility issues in distributed DRL with a new platform called Cleanba. Second, we build Gym-[mu]RTS, an efficient RTS testbed for conducting DRL novel research topics, such as game presentation designs and efficient learning techniques for dealing with invalid actions and sparse rewards. We also propose methods to scale agents to perform unit-level control in RTS games, lifting the artificial action space restriction of past works.