Accelerating Probabilistic Planning Research: High-Speed Implementations of Stochastic Markov Decision Processes and Their Catalogization
Robin Schmöcker, Alexander Dockhorn, Bodo Rosenhahn · IEEE Access · 2025
Probabilistic planning research, such as improvements to Monte Carlo Tree Search, is faced with two challenges. Firstly, one usually requires an algorithm’s evaluation on a variety of test problems, trying several hundred or even thousands of parameter combinations, which in turn requires a large number of models that can be efficiently queried. Secondly, one has to identify a suitable set of problems to do the evaluation on. This work is made up of two components that together provide a solution for both problems for the probabilistic planning community. The first component of ourwork is software, which provides numerous models that are implemented by hand with domain-specific optimizations for optimal runtime. In particular, the software component contains 22 parametrized, non-deterministic Markov Decision Processes (MDP), which is currently the largest open-source, non-simulator-based stochastic MDP library. The implementation provides access to methods required for planning (e.g. copying a state) as well as for reinforcement learning (e.g. getting an observation) through a C++ and a Python interface, which itself satisfies the Gymnasium interface. The second component of thiswork contains a detailed description as well as a statistical analysis of each model’s properties, which acts as a catalogization of the environments, enabling researchers to quickly select a set of environments satisfying their specific criteria.