Ensuring Reliability Through Combinatorial Sequence Coverage
Mohammad S. Raunak, D. Richard Kuhn, Raghu N. Kacker · IEEE reliability magazine · 2025
Errors in software and hardware systems do not always result from only a single set of input values. Instead, a series of input values or events often lead to failures. These sequences of events may or may not need to be contiguous. In either scenario, structural coverage metrics prevalent in traditional test selection approaches often fall short of exercising the code sufficiently to ensure the exercise of different sequences of inputs or events thoroughly to build confidence in its functions. We need to have metrics and measures to gauge how much of relevant combinations of input values and input sequences have been tested and verified for correct operation. Going beyond simple coverage of sequences in a test, value combinations over multiple tests in a sequence are also important for building highly reliable systems. Combinatorial sequence coverage and nonadjacent and adjacent ordered combination cover (OCCa) measures provide efficient means of achieving this type of verification. This article discusses combinatorial sequence coverage, ordered combination cover (OCC), OCCa, and their application and usefulness in runtime verification, vulnerability and fault detection, and reliability assurance.