GUI Element Detection via YOLOV8: A Deep Learning Approach for Widget Identification
S Benston Jose, Philip Samuel, Sumam Mary Idicula · 2024
Generating front-end code for mobile applications can be both time-consuming and repetitive. Many companies maintain individual UI design and development teams, where designers create the GUI, which is later converted to code by developers resulting in duplicated efforts. For firms focused on native app development, dual coding for Android and iOS UI screens is necessary. The main challenge of Graphical User Interface (GUI) automation is identifying the GUI widgets present in the user interface. Object detection, which is a computer vision technique, can indeed have a significant impact on addressing this challenge. Given the prevalence of text-based elements, employing the YOLO one-stage object detection algorithm is advantageous. Leveraging the pre-trained YOLOv8n algorithm facilitates custom element detection model creation. In this study 1,500 UI samples from the VINS dataset are employed for training, testing, and validation. This approach yields component identification accuracy surpassing 68%. Overall, this method accelerates code generation, minimizes redundancy, and enhances UI development efficiency.