An Empirical Study of Fault Localization on Novice Programs
Yuxing Liu, Jianying Chen, Jiamin Tang, Xiaoyi Tong, Liping Cai, Hengyuan Liu · 2024
Programming learning is becoming increasingly prevalent in college curricula, yet novices often encounter substantial difficulties in debugging due to their limited programming experience. In response to these challenges, automatic fault localization methods, such as Spectrum-Based Fault Localization (SBFL) and Mutation-Based Fault Localization (MBFL), have emerged as promising solutions. However, these methods are typically designed for industrial programs, which differ markedly from novice programs in terms of size and complexity. This discrepancy highlights a significant research gap in the application of these methods to novice programs. To address this gap, we conducted an empirical study to evaluate the fault localization performance and execution overhead of SBFL and MBFL in environments typical of novice programmers. Our research specifically examined how various program characteristics, including code coverage and mutation score, affect the accuracy of these localization methods. The study was comprehensive, involving experiments on 190 real novice faulty programs. The findings from our study demonstrate that both SBFL and MBFL are effective for fault localization in novice programs, though MBFL was notably more effective in our tests. MBFL demonstrated superior performance by accurately localizing 67, 96, and 114 faults within the${TOP}-{N} (N=1.\ 3.\ 5)$.