Prioritization of Crowdsourced Test Reports Based on Defect Severity and Frequency Weighting
YiHao Li, Lei Xiao, Weiwei Zhuang, Xiaozhu Xie, Jiawei Zhang · 2024
In crowdsourced testing of mobile applications, efficiently reviewing the large number of generated test reports has always been a significant challenge. Although many automated techniques have emerged in recent years, such as clustering, classification, and prioritization, which assist in automating the review process, existing methods often focus solely on fault detection rates, neglecting the severity and frequency of defects. Defect severity and frequency are critical indicators marked by users that can help developers prioritize problem resolution. We propose a novel multi-objective prioritization approach tailored for crowdsourced test reports, considering defect diversity, severity, and frequency. Initially, we assign weights based on defect severity and frequency to rank the reports. Then, we cluster the reports by combining both text and image features. Finally, the initial ranking is adjusted based on the clustering results to ensure broader error coverage and prioritize the most critical defects.We conducted experiments on datasets from industrial crowdsourced testing reports for three mobile application projects. We carried out experiments using datasets from industrial crowdsourced testing reports across three mobile application projects. The results indicate that, compared to existing methods, our approach detects all faults more quickly within a limited time, performs better than methods that consider only defect severity or frequency, and identifies a greater number of diverse high-severity errors earlier. These results validate the effectiveness of our method in improving testing efficiency and outcomes.