A Study of Threats to Validity in Controlled Software Engineering Experiments
Nils-Kristian Liborg · NORA - Norwegian Open Research Archives · 2004
This Masters Thesis describes the state of the art regarding threats to validity in controlled software engineering experiments. Among the 5453 articles published in 13 leading journals and conferences in the decade 1993-2002, 107 articles (2%) reported controlled experiments in which individuals or teams conducted one or more software engineering tasks. This thesis has a special focus on generalization regarding subjects, tasks, and environment (threats to external validity). I mainly look on two different aspects for each generalization type; if replications strengthen the validity and what kinds of arguments are used for generalizing or not generalizing. At the end of the analysis the raw data found for threats to internal validity are also summarized, but not presented in detail. The main result from this research is that most researchers are very vague in their conclusions regarding results of their experiments, and that they are to some degree ambiguous. Another result from this research is that researchers struggle with more or less the same problems regarding generalization of their results. The most important proposition from this thesis will be that researchers should write more precise discussions whether their results can be generalized or not.