The interrelations of features of questions, mark schemes and examinee responses and their impact upon marker agreement
Beth Black, Irenka Suto, Tom Bramley · Assessment in Education Principles Policy and Practice · 2011
In this paper we develop an evidence-based framework for considering many of the factors affecting marker agreement in GCSEs and A levels. A logical analysis of the demands of the marking task suggests a core grouping comprising: (i) question features; (ii) mark scheme features; and (iii) examinee response features. The framework synthesises several strands of previous empirical research. Additionally, we present new data, collected by observing the social dynamics of examiner meetings, to illustrate some effects of these interactions. We use our framework to argue that the two most important goals of research into marker agreement are prediction and control. That is, assessment agencies need to be able not only to predict the level of agreement between a marker’s mark and the ‘definitive’ mark, but also to anticipate the effects of different possible courses of action on the level of agreement. We conclude with a list of possible courses of action, considering their effect not only on marking accuracy, but also on overall validity – the overriding concern.