Test Prioritization with Optimally Balanced Configuration Coverage
Dusica Marijan, Marius Christian Liaaen · 2017
Testing configurable software for high assurancesystems developed in continuous integration requires effectivetechniques for selecting failure-inducing test cases, thoroughlycovering entire configuration space, while providing rapid feedbackon failures. This involves satisfying multiple objectives:maximizing test fault detection, maximizing test coverage ofthe configuration space, and minimizing test execution time, which often leads to compromises in practice. In this paper, weaddress this problem with a practical test optimization approachthat uses historical test data to determine an optimal order oftests ensuring high progressively uniform configuration coverage, early fault detection, and rapid test feedback. We extensivelyvalidate the approach in a set of experiments using industry testsuites, and report experimental results showing the improvementin efficiency compared to industry practice. In particular, theapproach showed to increase the uniformity of configurationcoverage by 39% on average, which increases fault detectionup to 15%, while just slightly delaying test feedback.