A Heteroscedastic Bivariate Distribution Arising from a Model for Rater Agreement, and its Fitting by Simulation
Timothy Paul Hutchinson · SSRN Electronic Journal · 1998
The starting point is a dataset showing that two raters, judging proficiency in spoken Russian, appear to disagree more about the relatively expert speakers than about the novices. A bivariate distribution is invented and shown to fit the data better than the bivariate normal distribution does. The chief features of thedistribution are that it is a variables-in-common model, true score plus error for each rater, and that the scatter of error is greater when the true score is high than when it is low. The method of fitting the distribution to the data is simulation. Accordingly, an explicit expression for the joint distribution of the two observed scores is not required. The software used has ranking and recoding commands of one line each, so it is easy to ensure the fitted marginal distributions exactly match the data, and it is unnecessary to estimate parameters representing the boundaries between the grades of rating.