Vote-Counting for Combining Quantitative Evidence from Empirical Studies- An Example
Parastoo Mohagheghi, Reidar Conradi · 2006
The amount of empirical evidence on various software engineering methods is small but growing. As a result, more attention is paid to collection of evidence and combining the results of empirical studies. Three methods are discussed in literature for combining quantitative evidence obtained from empirical studies: combining p-values, meta-analysis and votecounting. We describe our experience with the vote-counting method applied on the results of a review on software reuse. This method was selected since the reviewed literature did not provide enough information to apply the other methods, while the statistical or practical significance was often discussed. In votecounting, the numbers of studies showing significant beneficial and harmful effects are counted. We adapted the method to include scale of the studies, both statistical and practical significance, and also show positive and negative results when significance was not discussed or significant results were not found. The method can be applied as a first step to combine evidence by summarizing the results and identifying where there are gaps in empirical research, but cannot answer questions on the magnitude of effect.