Editorial: Testing in the large through the small?
Robert M. Hierons · Software Testing Verification and Reliability · 2003
Testing in the large through the small?We are frequently told that computer systems are becoming increasingly complex.Assuming this is the case, does this make testing more difficult?We are also told that computer systems are becoming increasingly significant.Does this make it even more important that our testing is effective?If the answer to both of these questions is 'yes' then we seem to have a real problem.This might help explain the results of a recent study by the National Institute of Standards and Technology [1].This concluded that the cost to the U.S. economy, of poor testing, was in the order of 59.5 billion dollars a year.I am sure that such problems are not restricted just to the U.S.!So, what is the solution?If I had the answer to this question it is just possible that I would currently be in rather more luxurious surroundings.However, I hope that the contents of this issue point to one approach that can help and illustrates links between a number of approaches.This issue's three papers relate to a theme: how can we test 'large/complex' systems by testing 'small/simple' systems?Naturally, this is an important issue-system complexity continues to increase and we all know that many of our favourite testing/verification techniques struggle beyond some level of complexity.This problem is exacerbated by the development of distributed state-based systems: here the number of states of the overall system grows exponentially as the number of components increases.The papers all approach this problem by using some of the scientist's favourite tools.Specifically, they use at least one of abstraction (abstract away details to simplify the problem) and reductionism (break the problem into sub-problems and tackle these).Reductionism is applied through the identification and testing of components.In their paper entitled 'Test suite minimization for testing in context', Anido et al. consider the problem of testing a single component embedded within a context that is known to be correct.Thus, rather than considering the testing of the entire system, they look at the testing of the single component (it is possible that the user/environment cannot send test input directly to this component and cannot receive test output directly from this component).They show that a given test suite may be reduced by considering how each test sequence within it contributes to the testing of the suspect component.Assuming the context is correct, this process is guaranteed to preserve the test suite's effectiveness.By contrast, Rusu, in his paper entitled 'Combining formal verification and conformance testing for validating reactive systems', extracts components from the specification and tests on the basis of these.The papers of Anido et al. and Rusu have another common element: compositionality.So, how do these papers apply abstraction?De Francesco and Lettieri, in their paper entitled 'Checking security properties by model checking', show that a program may be simplified, through abstraction, if we want to verify that certain security problems do not exist.Once this abstraction has been applied, model-checking techniques are used, through the Symbolic Model Verifier (SMV) tool, to determine whether these security problems exist.Importantly, if the abstracted model does not have