Managing evaluation goals for training
John M. Carroll, Mary Beth Rosson · Communications of the ACM · 1995
valuations of training material serve many goals and are characteristically underresourced.In this article, we describe a framework for managing training evaluation in such contexts.We illustrate by example the feasibility of taking a broad approach toward training evaluation.Scriven [21] introduced the terms formative and summative in his classic work on the objectives and processes of instructional evaluation.Formative evaluation seeks to identify aspects of a design that can be improved.A typical implementation would be having users think aloud as they work through a tutorial to produce a real-time commentary on the nature of the learning experience.Each comment is tightly coupled to a specific design feature, facilitating the identification of problems and reasoning about design solutions.Summative evaluation seeks to gauge a design product.A typical implementation would be measuring time-on-task and error rate to guide decision-making about whether a training system (or the software it supports) should be produced and marketed, or purchased.Formative evaluation work succeeds to the extent that it identifies priorities for redesign and refinement, and all the more to the extent that it provides concrete guidance as to how that redesign and refinement should be executed.Summative evaluation work succeeds to the extent that it allows a design product to be located on various measurement scales with respect to other design products; the stronger the scale (i.e., ratio scale versus interval scale versus ordinal scale versus nominal scale), the better the summative evaluation.Scriven [21] also distinguished two basic types of evaluation method, intrinsic and pay-off : "If you want to evaluate a tool, say an axe, you might study the design of the bit, the weight distribution, the steel alloy used, the grade of hickory in the handle, etc., or you might just study the kind and speed of the cut it makes in the hands of a good axeman."The examples discussed---thinking-aloud commentaries and task time/error rate measures---are both pay-off evaluations: empirical observations of the character and performance concomitants of training.