Leveraging Execution Trace with ChatGPT: A Case Study on Automated Fault Diagnosis
Takafumi Sakura, Ryo Soga, Hideyuki Kanuka, Kazumasa Shimari, Takashi Ishio · 2023
ChatGPT possesses the ability to identify potential causes of bugs in a program, which can be used for fault diagnosis. Although ChatGPT cannot always provide accurate responses, it can provide the confidence level that is trained to be correlated with the accuracy of the response, and the confidence level helps verify the accuracy of responses. Through preliminary trials, we found that the explanatory power of potential causes and the confidence level became low even for accurate responses when runtime information is needed to identify the causes of bugs. In this study, we propose a method to construct prompts based on the target program and its execution traces to improve the accuracy of fault diagnosis by ChatGPT. Through case studies using five bugs from Defects4J, we obtained the following two results: (1) For four bugs, the explanatory power of the responses to potential bug causes improved using the information contained in the execution traces. (2) For three bugs, the confidence level was higher for the accurate responses when execution traces were available than when they were not. These results suggest that by using program execution traces in prompts, the accuracy of fault diagnosis by ChatGPT can be improved.