Variability of referees' ratings of conference papers
Henry M. Walker, Weichao Ma, Dorene Mboya · 2002
The authors developed an on-line system to automate the paper submission and reviewing process for SIGCSE 2000. This system provided a mechanism for a statistical study of the ratings of papers by different referees. To compile sufficient sample data, 10 papers were sent to about 100 referees each, in addition to the normal reviewing of the 220 submitted papers. This paper reports the results of this work, based on 1917 reviews completed by 482 referees. The data support correlations of a paper's overall rating with internal factors (technical content, writing quality, originality, significance) and external factors (referee gender, nationality, familiarity with topic, etc.). The study also shows the effect of group reviews (similar to NSF panels), and the divergence of opinions between authors and referees concerning a paper's subject.