CAMETRICS: A Tool for Advanced Combinatorial Analysis and Measurement of Test Sets
Manuel Leithner, Kristoffer Kleine, Dimitris E. Simos · 2018
Combinatorial testing (CT) has established itself as an efficient and effective approach for generating test sets, guaranteeing that all interactions of parameters up to a given strength t are covered. While numerous publications in recent years have addressed the problem of generating Covering Arrays (CAs), the central mathematical construct used in CT, as well as measuring various aspects of these test sets, there is a distinct lack of usable and automated means to derive these measurements and thus determine the combinatorial quality of a test set and evaluate the behavior of the algorithm that produced it. We therefore introduce cametrics, a tool to compute various combinatorial coverage measures of a given test set, distribution of parameter value combinations, i.e. how the different t-tuples are distributed, as well as distance metrics and coverage gain per test. Such combinatorial analysis of test sets is crucial also for practitioners to estimate the residual risk that remains after testing by revealing untested parameter interactions. The tool is available as a web-based application allowing users to easily visualize the combinatorial metrics computed for test sets. Our experimental evaluation and the feature set of cametrics compares favorably to existing tools in the field.