Towards an AI playing Touhou from pixels: a dataset for real-time semantic segmentation
Dario Ostuni, Ettore Tancredi Galante · 2021 IEEE Conference on Games (CoG) · 2021
When playing from pixels, AIs share some of the struggles that humans face when playing a game, namely not knowing its internal state. In this paper we begin the exploration of the AI-playing-from-pixels problem for Touhou, a bullet hell game. Albeit being a massively popular game in some niches, the community has yet to produce an AI capable of beating it, while looking only at pixels, in Lunatic mode, the hardest difficulty. We propose, as a first step, to build a semantic segmentation model to create a bridge to the internal-state-looking AIs. To achieve this, we created a dataset to train models for this task. This dataset is procedurally generated using manually labeled assets from classic era Touhou games. After selecting five state-of-the-art real-time semantic segmentation networks, we trained them using our dataset. The results indicate that the models produced have a high classification performance over the validation set. However all models, but one, are too slow to run in real-time at the game's target frame rate. On real game footage the models show promising results, but the dataset needs to be strengthen to account for noise sources in the real game.