Dataset generation and model fine-tuning for GUI anomaly detection
Yadini Pérez López, Juan David Ochoa Saavedra, Marcio Modesto Da Silva, Matheus Lima Titon, Erison Correia De Oliveira Neto, Guilherme Cruz Amarante Arantes, Fabio H. Oliveira · 2024
In the rapidly evolving smartphone industry, ensuring the quality and usability of mobile applications is crucial, particularly through robust Graphical User Interfaces. Traditional manual GUI testing methods no longer support the complexity of modern applications. This paper presents a methodology for GUI anomaly detection leveraging synthetic data generation and deep learning. Our approach addresses the limitations of existing methods by creating a dataset that accurately replicates real-world GUI anomalies, such as cropped text, overlapping elements, pixelation, and null values. We used the dataset generated to train and evaluate the well-known YOLOv8 model, attaining 97% mAP to recognize the anomalies. Our study underscores the potential of automated, scalable solutions for automated enhancing GUI testing.