Missing Threats: Dealing with the Treatment-sensitive Factorial Structure Bias in Empirical Software Engineering
Sabato Nocera, Giuseppe Scanniello · 2025
Researchers in Software Engineering (SE) frequently use program analysis tools (or simply tools) as part of their experiments. These tools can be used to measure software attributes (response/dependent variables) or as part of an intervention (treatment/independent variable). But what happens when researchers use the same tool both as a measurement instrument and as part of an intervention? They might be dealing with a threat to construct validity: the treatment-sensitive factorial structure bias. Accordingly, the participants receiving the intervention might experience what is being evaluated in the experiment differently from the other participants. This means that the results of the study might favor one group of participants over others based on their exposure to the intervention (which involved the tool used to make measurements). In this paper, we present a reflection-based analysis of the treatment-sensitive factorial structure bias. We share suggestions on how to mitigate this threat in SE studies (e.g., experiments or cohort studies). We advise researchers to use measurement instruments not associated with the treatment. In case this is not possible, the association between the measurement instrument and the treatment should be as small as possible. Also, choosing a proper version and/or configuration of the measurement instrument might mitigate the treatment-sensitive factorial structure bias.