An axiomatic theory of software test data adequacy criteria
Stuart Zweben, Allen S. Parrish · 1990
In software testing, various criteria have been proposed that assess when a set of test data sufficiently tests a program. An example of such an is statement coverage, which requires that any adequate test set cause every statement in the program to be executed. Testing researchers have long recognized the need for developing a solid theoretical foundation for the discussion and analysis of existing criteria, as well as for the development of new criteria. Previous research has resulted in two important contributions toward the development of such a theory: (1) a framework that includes the formal definition of a criterion, and (2) a set of properties that are intended to fundamentally characterize useful criteria under certain assumptions about the testing process. These properties of criteria were deliberately informal, in order to facilitate their acceptance in the practicing software engineering community. In this thesis, we formalize the properties and integrate them into the previously proposed framework, with the goal of developing a formal axiomatic theory of adequacy criteria. In developing our axiomatic theory, we appeal to principles from traditional axiomatic systems in mathematics. Among other formally desirable characteristics, we require that our selection of properties be consistent and independent. This integration reveals both inconsistency and dependencies among the previously proposed properties. We show how a slight modification to the assumptions removes the inconsistency, and we extend the original framework to allow such assumptions to be expressable. The extended framework allows us to more accurately reflect the scope of programs, specifications and test sets within which a particular criterion is defined. We develop an independent set of properties within this extended framework, for the types of criteria to which the original properties were meant to apply. We then show how our extended framework allows the development of related sets of consistent, independent properties that apply to a more general class of criteria. We also show how weak properties within the sets can be strengthened while preserving the consistency and independence of the sets. We apply our theory to the resolution of various issues that have been glossed over in the literature. The theory clarifies the distinction between test methods (which generate supposedly useful test sets for a given program and specification), and adequacy criteria (which are used as a basis for deciding if enough testing has been done). We show how this distinction effects the appropriateness of certain properties. We also apply some of the properties to the previously proposed data flow criteria, revealing new results concerning the appropriateness of these criteria in the context of certain assumptions about the testing process.