Evaluating the interrater agreement of process capability ratings
Pierfrancesco Fusaro, K. El Emam, B. E. Smith · 1997
The reliability of process assessments has received some study in the recent past, much of it being conducted within the context of the SPICE (Software Process Improvement and Capability dEtermination) trials. In this paper, we build upon this work by evaluating the reliability of ratings on each of the practices that make up the SPICE capability dimension. The type of reliability that we evaluate is inter-rater agreement: the agreement amongst independent assessors' capability ratings. Inter-rater agreement was found to be generally high. We also identify one particular practice that exhibits low agreement in its ratings.