Incentivizing Effort and Precision in Peer Grading
Anujit Chakraborty, Jatin Jindal, Swaprava Nath · arXiv (Cornell University) · 2018
Inspired by the easily scalable solution of peer evaluation in various domains that need an ever-growing body of material to be evaluated, e.g., in journal submissions, or in evaluating proposals for public funding, peer-grading has found widespread use in Massive Open Online Coursewares (MOOCs). Current peer-grading practices often rely on the goodwill of the peer evaluators for accuracy and fairness in the evaluation, while evaluators can be competitive or energy-saver. In this paper, we consider strategic graders and introduce a mechanism, TRUPEQA, that (a) uses a constant number of instructor-graded answer-scripts to quantitatively measure the accuracies of the peer graders and adjusts the finally assigned scores accordingly, and (b) penalizes deliberate under-performing. We show that this mechanism is unique in its class in satisfying certain properties desirable of a peer-grading mechanism. When the cost of evaluation is factored in, a modification of TRUPEQA implements the social optima in an equilibrium. We also run a peer-grading experiment in a classroom environment and find evidence that the final assigned grades are closer to the true grades under TRUPEQA than under the peer-grading mechanism currently used in popular MOOCs.