A Family of Experiments to Investigate the Influence of Context on the Effect of Inspection Techniques
Marcus Ciolkowski, Forrest J. Shull, Stefan Biffl · 2002
For a growing population of researchers in software engineering, empirical studies have become a key approach of research. Empirical studies may be used, for example, to evaluate technologies and help to direct further research by revealing what problems and difficulties people have in practice. Without empirical studies, we have to rely only on intuition or educated opinion. Individual empirical studies often yield interesting results for their particular context, but typically this context is not described in sufficient detail to decide whether another context is similar enough to apply the conclusions of the study also there. We argue instead that families of experiments with a common framework for collecting context data are necessary in order to abstract conclusions at a useful level of detail. This paper describes a method to plan, conduct, and analyze coordinated, or concerted, families of experiments. The goal of the method is to maximize the quality and benefit of the individual empirical studies as part of the family and to minimize the effort for researchers by reusing experiment know-how. This is achieved by providing, for all studies of the experiment family, a common framework for context measurement, study preparation, material, and analysis. We apply the method to describe the planning steps for an experiment family on the influence of context on the effectiveness of defect reduction techniques. We focus on a particular technology, reading techniques for inspections, to instantiate this work. The first step of this experiment family is a broad survey of software companies on the state of the practice of inspection process and inspection techniques. The second step is to benchmark state-of-the-art inspection techniques with the participating organization's own documents and inspection techniques.