The Effect of Scoring Order on the Independence of Holistic and Analytic Scores
Nancy Robb Singer, Paul G. LeMahieu · eScholarship (California Digital Library) · 2011
Conventional wisdom and common practice suggest that to preserve the independence of holistic judgments, they should precede analytic scoring.However, little is known about the effects of scoring order on the scores obtained or if true holistic scoring is even possible from the mind of a scorer who has already been trained to and will be asked to provide analytic scores as well.This research explores the matter of independence of scores and the effects of scoring order upon those judgments.Our analysis shows statistically significant differences in mean scores under the two conditions (holistic scoring preceding analytic and the reverse), with the holistic scores more nearly replicating "pure" holistic scoring only when it precedes the analytic.This research affirms that when readers will be asked to score both ways, holistic scoring should precede analytic scoring.It also suggests interesting insights into the cognitive processes engaged by scorers as they score holistically and analytically.When assessing student writing, there are often legitimate reasons for seeking both a holistic score as well as a set of analytic scores.Holistic scoring refers to assessments in which a single summary judgment of quality is rendered, albeit often guided by a conceptual framework that articulates essential dimensions upon which quality is to be defined (Huot, 1988;White, 1984).In analytic scoring, writing is described in terms of essential attributes and individual judgments recorded for several constituent attributes.Holistic scoring is thought by some to enable a more complete and appropriate depiction of a phenomenon as complex as writing, which they feel can never be adequately deconstructed into several parts (White, 1994).Moreover, singular summary judgments may fulfill certain assessment purposes perfectly well, while being quicker and, therefore, more economical (Spandel & Stiggens, 1980).Analytic scoring, on the other hand, may be considered more informative and therefore potentially relevant to instructional and programmatic decision making, though it is more time consuming and therefore generally more expensive to conduct (Cooper & Odell, 1977;Rabianski, 1979).