Human-like UI Automation through Automatic Exploration
Xudong Li, Dajun Zhou, Like Zhang, Yanqing Jing · 2020
Most UI testing tools for mobile games are designed to help us create and run the test cases with scripts. However, these scripts must be manually updated for new test cases, which increases the test cost. In this paper, we propose a method to implement humanlike UI automation through automatic exploration in mobile games. Our method can automatically explore most UIs by recognizing and operating the UI elements similar to manual UI testing. First, we design a lightweight convolutional neural network to detect the buttons in the UI image captured from the mobile phone. Next, we build a directed graph model to store the visited UIs during automatic exploration. Finally, according to our exploration strategy, we choose one button from the UI image and send a click action to the mobile phone. Our method obtains over 85% UI and button coverage rates on three popular mobile games.