Raters and rater training
Slobodanka Dimova, Xun Yan, April Ginther · 2020
This chapter discusses the process of rater selection, initial training, and continuous norming. It addresses the different methods of rater training and the importance of rater agreement for the reliability of test results and the different instruments and methods that can be employed for the analysis of rater behavior and inter- and intra-rater reliability. In local testing, be it for placement, screening, or certification purposes, the raters are usually language teachers. An important advantage of rater-mediated assessments is that raters can use their prior knowledge, expertise, and experience to assist with scoring. The involvement of teachers as content experts or raters who are familiar with the local contexts can bring both benefits and challenges to the scoring process. Consensus estimates reflect the percentage of agreement between the scores assigned by two or more raters. The chapter concludes with an overview of best practices for rater training, including rater cognition, benchmarking, establishing a rater group as a community of practice.