Specification-Based Testing in Software Engineering Courses
Gene L. Fisher, Corrigan Johnson · 2018
In 2016 we introduced a tool called "Spest// for the automated generation of software tests from formal specifications. That introductory presentation of Spest described its basic functionality and our initial results of using Spest in software engineering courses. Here we describe further experience of using Spest in classes, including qualitative and quantitative analyses of its effectiveness. The analysis consists of a qualitative survey of students/ experience, a quantitative study of the readability of Spest-generated tests, and a quantitative analysis that compares the coverage quality of hand-written student tests with Spest-generated tests. The results of the analyses are mixed. The experience survey finds that a majority of students did not enjoy using Spest nor fully understand how to use it effectively. The results of the readability study show that Spest-generated tests are not as readable for students as tests written by human experts, however the differences in readability are not extreme. Finally, the results of the coverage comparison are good, showing that Spest-generated tests achieve better code coverage than students/ hand-written tests. Given the mixed results, we discuss how we are moving forward to make Spest a more usable and effective tool.