Coefficients of Agreement for Criterion-referenced Tests with Missing Data
Jin‐Uk Kim · Journal of Sport and Leisure Studies · 2021
Purpose: The purpose of this study is to analyze coefficients of agreement used as an evaluation tool for criterion-referenced tests. In common situations where more than three raters and more than three categories with missing data, we propose a method of easily calculating coefficients of agreement using vectors and matrices and modify the existing method of calculating the coefficients. Method: The data was summarized by raw data vectors, and by using them, two contingency tables for the subject by category and for rater by category were created. Results: The contingency tables were then used to calculate four frequently used coefficients of agreement, S (Benett et al., 1954), π (Scott, 1955), κ (Cohen, 1960), and AC1 (Gwet, 2002). Conclusion: By clearly defining different chance agreements assumed in each coefficient of agreement, an appropriate calculation could be made and a more valid coefficient of agreement could be obtained.