Response to Livingston's Comment on Minimal Beardedness.
GLENN L. ROWLEY · Journal of Educational Measurement · 1982
Livingston (1982) has made the interesting suggestion that the advice tendered to the Minimal-Beardedness Testing Program (Rowley, 1982) by our Expert on StandardSetting (ESS) may have been faulty. In particular, Livingston has argued that ESS gave bad advice with respect to the use of the Contrasting Groups Method. Instead of seeking out clearly bearded and clearly unbearded subjects, it would have been more appropriate to obtain judgments of beardedness from students whose test scores were not at the extremes. Clearly Livingston is concerned (as we were) that we should actually have access to data from the region on the scale at which the important decisions are going to be made. He concluded that the Contrasting Groups Method, as described by ESS, was an inappropriate method of setting standards on the Minimal-Beardedness Test, and, by implication, on any other test. I have no quarrel with this conclusion, since it coincides with my own. But did ESS provide us with an inaccurate account of the method? Tempting as it is to blame all our woes on ESS, I am unable to do so. Although the Contrasting Groups Method has since been widely publicized (e.g., Shepard, 1980, p. 454), its original formulation is attributable to Zieky and Livingston (1977). Table 1 draws some contrasts between the Contrasting Groups Method, as originally described by Zieky and Livingston, and Livingston's (1982) account of it. It is clear from Table 1 that the Contrasting Groups Method has been undergoing fairly rapid evolutionary change. In the 1977 version, we are required to obtain judgments only on those students who are clearly masters or clearly non-masters; for the 1982 version we need judgments on the whole sample. The 1977 version requires us to obtain a representative sample of students; for the 1982 version that is not important although the requirement is a much tougher one-the students sampled at each score level must be representative of all students with test scores at that level. And, finally, in 1977 we were to base our standard-setting only on those students about whose status judges were certain. By 1982, this had become an all-too-common mistake. I wonder why? I am advised by ESS that Livingston's (1982) formulation is superior to that by Zieky and Livingston (1977) since the decision about the choice of a standard is actually based on data from subjects whose scores fell within that region. ESS has expressed reservations about the use of the term to describe a classification based on the test score when it does not agree with that of a judge. ESS argues that the use of the test is intended to provide us with more precise information than is available to the judge, and that disagreements might well be attributed to error on the part of the judge. In the case of the Minimal Beardedness Program it was the professed inability of the judges to make that classification with confidence which led us to consider the use of a test. And ESS points out that Livingston's (1982) method does not require the use of contrasting groups, so perhaps a new title might be appropriate. In one respect, of course, Livingston is correct. ESS has been dispensing bad advice, and I have scolded him severely for advising me to try the Contrasting Groups Method, as