Survey of Controlled Software Engineering Experiments with Focus on Subjects
Ove Hansen · NORA - Norwegian Open Research Archives · 2004
Empirical research within the Software Engineering field is a fairly new discipline. This calls for the development of new methods and new standards for how to perform and report experimental procedures and results. For research to be of interest for the community we need to know how the research has been performed, and within which context. Through the CONTEXT project, employees and M.Sc. students at Simula Research Laboratories have performed a survey on all controlled experiments reported in 12 leading journals and conferences during the period 1993 to 2002; a total of 118 experiments from 107 papers. Our aim has been to find out how the experiments are reported – what information is supplied, and to some extent the quality of this information. The focus of this thesis is on the subjects participating in the experiments. What information is provided regarding who they are and their backgrounds, and to what extent variables that might be of relevance for the validity and generalizability are described. This gives an indication of the maturity of this research. Almost all controlled experiments in our field are aimed at the professional population, yet only 26 % of the experiments in our survey used professionals as subjects. Only 19 % of the experiments using students as subjects generalize their findings to professionals. The heterogeneity of the subjects is generally not given much attention, as differences between them are rarely discussed and background variables often described in little detail. We have found that whether or not it is appropriate to use students as subjects is highly dependent on the issue studied and which populations the research is targeting. The terms “student” and “professional” are in many cases misleading, as these may not be relevant for the task at hand. It may often make more sense to talk about novices and experts. The overall impression of controlled experiments in the SE field is that this discipline is developing, but not yet mature. Our analysis shows that there is a need for standards and guidelines that authors should adhere to when publishing their work. Today many papers provides so little information that it is difficult for the reader to get an impression of to what extent the results are valid, and if they are – to whom.