Improving expert prediction of issue resolution time
Dietmar Pfahl, Siim Karus, Myroslava Stavnycha · 2016
Predicting the resolution times of issue reports in software development is important, because it helps allocate resources adequately. However, issue resolution time (IRT) prediction is difficult and prediction quality is limited. A common approach in industry is to base predictions on expert knowledge. While this manual approach requires the availability and effort of experts, automated approaches using data mining and machine learning techniques require a small upfront investment for setting up the data collection and analysis infrastructure as well as the availability of sufficient past data for model building. Several approaches for automated IRT prediction have been proposed and evaluated. The aim of our study was (1) to compare the prediction quality of expert-based IRT prediction in a software company located in Estonia with that of various fully automated IRT prediction approaches proposed and used by other researchers, including k-means clustering, k-nearest neighbor classification, Naïve Bayes classification, decision trees, random forest (RF) and ordered logistic regression (OLR), and (2) to improve the current IRT prediction quality in the company at hand. For our study, we analyzed issue reports collected by the company in the period from April 2011 to January 2015. Regarding our first goal, we found that experts in the case company were able to predict IRTs approximately 50% of the time within the range of ±10% of the actual IRTs. In addition, 67% of the experts' predictions have an absolute error that is less or equal 0.5 hours. When applying the automated approaches used by other researchers to the company's data, we observed lower predictive quality as compared to IRT predictions made by the company's experts, even for the best-performing approaches RF and OLR. Regarding our second goal, after unsuccessfully experimenting with improvements to the RF and OLR based approaches, we managed to develop models based on text analysis that achieved a prediction quality at par or better than that achieved by company experts.