Leveraging ChatGPT to Predict Requirements Testability with Differential In-Context Learning
Mahima Dahiya, Rashminder Gill, Nan Niu, Hemanth Gudaparthi, Zedong Peng · 2024
Testability is a desired property of requirements, indicating how easy or difficult a requirements artifact supports its own testing. Prior work predicts natural language (NL) requirements’ testability by training a decision tree (DT) via some readability and word measures. To explore better ways of predicting requirements testability, we examine in this paper large language models-ChatGPT in particular. Our experiments on a total of 1,181 requirements from six software systems show that ChatGPT’s zero-shot learning performs worse than the DT. A main reason is due to the lack of context specific to the testability prediction task. However, applying ChatGPT’s incontext learning (ICL) reveals a limitation of skewed examples caused by the imbalanced data. Thus, we propose a novel approach, called differential ICL, to address the challenges by exploiting the DT and show quantitatively the higher accuracy achieved by differential ICL.