Editorial: Validating our findings
Robert M. Hierons · Software Testing Verification and Reliability · 2005
Validating our findingsIf we come up with a new approach such as a new testing technique how do we know that it is any good?In other words, how do we evaluate our research?This is a surprisingly controversial topic in software testing and, indeed, in software engineering as a whole.One of the many nice things about this issue of STVR is the variety of evaluation methods used.At one extreme, the authors of the first paper formally define the problem they are aiming to solve, give two new testing methods and evaluate these by proving that they have the required properties.Naturally, a strength of this theoretical approach is that (assuming the proofs are correct!)we know that the results always hold.However, many other problems are not amenable to this method of evaluation and our other papers both use one of the main alternatives: empirical investigations.Of particular note here is the third paper whose authors replicate a previous study using a different sample of programs.It is extremely encouraging to see this: replication of experiments is common practice in the physical sciences but has been much less usual in software engineering.Hopefully this will become a trend and we will see many more replicated experiments in the future.Turning in more detail to the contents of this issue, the first paper, by Manuel Núñez, Ismael Rodríguez and Fernando Rubio, considers the problem of testing the behaviour of an e-commerce agent that has been specified as a 'utility state machine'.A utility state machine is like a standard extended finite state machine with the addition of time and a utility function for each state.In essence, the utility function for a state represents what the agent is trying to achieve in this state; when in this state, the agent can exchange resources with other agents if such exchanges increase its utility.State transitions occur under specified conditions and can be seen as moving the agent to a new state representing a different objective when the current objective is achieved.The authors describe two forms of testing such an agent: active testing in which we introduce a tester as another agent to interact with the agent being tested; and passive testing in which we observe the behaviour of the agent acting in its context.To complement the formal validation mentioned in the first paragraph above, the authors have also presented a case study whereby the formalism was applied to a system called Kasbah: a pioneering multi-agent e-commerce system for buying and selling goods.The second paper, by Daniel Hoffman, Paul Strooper and Sarah Wilkin, describes a new tool, JUnitDoc, which is based on the tools JUnit and Javadoc.The key idea is that Javadoc comments are used in order to embed test cases in the code.These test cases assist in documenting the code since they are examples of expected behaviour and Javadoc can be used to extract them as HTML documentation.They can also be extracted in order to produce JUnit test drivers.The authors suggest that test cases are placed as high as possible in the inheritance hierarchy in order to generate a test hierarchy that mirrors the inheritance hierarchy.A detailed example demonstrates the benefits for testing and, in addition, a small controlled experiment was conducted to compare the readability of the JUnitDoc documentation to formal specifications written in Object-Z.The results of this evaluation