Experimental comparison of approaches for checking completeness of test suites from finite state machines
Adilson Luiz Bonifácio, Arnaldo Moura, Adenilso Da Silva Simao · Information and Software Technology · 2017
Many approaches have been proposed for checking test suite completeness for Finite State Machines (FSMs). Some approaches provide sufficient conditions whereas others give necessary and sufficient conditions for test suite completeness. One method, called the CONF method, is based on sufficient conditions, and relies on a search for confirmed sets when checking completeness. If a confirmed set cannot be found, then the outcome is inconclusive. Another method, the SIM method, is based on the notion of simulation relations, and relies on necessary and sufficient conditions when checking test suite completeness. The SIM method always returns conclusive verdicts about suite completeness. In this work, we describe experimental results comparing these two methods. We also investigate when both methods can be combined for checking completeness of test suites. We evaluate both strategies according to different parameters of the FSMs, such as the number of states and the number of transitions in the FSM models, the size of input and output alphabets of the FSM models, as well as the size of the test suites. We also report on the relative rates of conclusive and inconclusive verdicts when using both methods. We see that these methods are complementary, which allows for a combined strategy: the CONF method is the fastest in terms of processing time, while the SIM method is not as scalable in terms of the size of the specifications. The experimental results indicated a substantial difference for the rate of positive verdicts obtained by the SIM method when compared with the number of positive answers returned by the CONF method.