Discrepancy discovery in search-enhanced testing
Werner Janjic, Florian Barth, Oliver Hummel, Colin Atkinson · 2011
Automating software testing can significantly reduce the time and effort required to assure the quality of software systems, and over recent years significant strides have been made in test automation techniques. However, one aspect of software testing that has always resisted full automation is the determination of the expected results for given system states and input values - the so called "oracle problem". Fortunately, the recent advent of a new generation of software search engines containing millions of reusable software artifacts offers an elegant solution to this dilemma. Once a search engine is able to deliver multiple results that conform to a given specification (by searching for and adapting preexisting components), multi-version testing of software with "harvested" oracles becomes a feasible alternative to manual oracle definition. In this paper we present an approach to Search-Enhanced Testing with a focus on the discovery of discrepancies between the results returned by harvested test oracles and a Component Under Test for randomly generated test invocations. Our current research focuses on validating the hypothesis that human test engineers will find more defects when analyzing such automatically discovered discrepancies than when developing test cases using traditional coverage criteria.