Automatic Item Generation via Frame Semantics: Natural Language Generation of Math Word Problems.
Paul Deane, Kathleen M. Sheehan · 2003
This paper is an exploration of the conceptual issues that have arisen in the course of building a natural language generation (NLG) system for automatic test item generation. While natural language processing techniques are applicable to general verbal items, mathematics word problems are particularly tractable targets for natural language general techniques. The problem is to have a method for representing verbal content that can play a dual role, serving both as a representation of the generic conceptual structure that appears to be critical for mathematics word problems and that can provide the basis for natural language generation. An excellent candidate for this purpose can be found in Frame Semantics, a linguistic theory of word meaning that appears to have exactly the needed properties. Each type of mathematics word problem appears to make systematic use of a specific semantic frame. Distance-ratetime problems provide an excellent illustration of the principles involved. The analytical methods outlined in this paper have been used as the basis for the construction of a prototype NLG, Model Creator, that supports the automatic item generation of distance-rate-time problems. The paper describes the prototype system and notes its potential. The real power of NLG for automatic item generation will come when language resources can be relinked and reused, but even it its current form, it offers considerable power and flexibility for generating new items in a controlled manner. (Contains 68 references.) (SLD) Reproductions supplied by EDRS are the best that can be made from the original document. PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS BEEN GRANTED BY TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) 1 U.S. DEPARTMENT OF EDUCATION Office sf Educahonal Research and Improvement EDUcATIONAL RESOURCES INFORMATION CENTER (ERIC) This document has been reproduced as received from the person or organization originating it. 0 Minor changes have been rnada to improve reproduction quality. Points of view or opinions stated in this document do not necessarily represent official OERI position or policy. Automatic Item Generation via Frame Semantics: Natural Language Generation of Math Word Problems Paul Deane Educational Testing Service Princeton, NJ 08541 Kathleen Sheehan Educational Testing Service Princeton, NJ 08541 Paper Presented at the: Annual Meeting of the National Council of Measurement in Education Chicago, ILApril 2003 BEST COPY AVAILABLE 2 1. The Role of Natural Language Processing in Automatic Item Generation 1.1. The Scope of Automatic Item Generation There is an increasing interest in the use of Automatic Item Generation (AIG) in educational assessment, concomitant with the development of technologies which have brought delivery of test content by computers into the mainstream. Early work (e.g. Bejar 1986, 1993, 1996, 2002; Bejar & Yocom 1991, Hively, Patterson & Page 1968, Irvine, Dunn & Anderson 1990, Laduca et al. 1986, Meisner, Luecht & Reckase 1993) has led to a flowering of AIG research (see, e.g., Irvine & Kyllonen 2002 for an overview.) Automatic item generation -the practice of creating assessment items algorithmically -can be motivated in part by a number of obvious practical advantages (cf. Bennett in press). AIG can speed or even partially automate the development of new items, it can provide similar items at the same level of difficulty, thus improving test security, and it can support adaptive testing by providing similar items that vary systematically in difficulty. Given the attractiveness of these goals, much recent research has focused upon evaluating the extent to which these promises can be met (cf. Bejar et al. 2002, Enright, Morley & Sheehan 1999, Enright & Sheehan 2002, Hombo & Dresher 2001, Wright 2001). However, automatic item generation obviously and critically depends upon well-founded models of the cognitive abilities underlying performance. Where such models are lacking, generation algorithms can have heuristic usefulness only. Much of the recent progress in AIG has come in areas where there are well-founded cognitive theories to support the development of AIG algorithms, e.g., matrix completion (Embretson 1993, 1998, 1999, 2001) or analytical reasoning (Newstead et al. 2002). It is thus important to approach AIG in the light of theories of test construction which ground item design upon an evidentiary basis, such as Evidence Centered Design (ECD), cf. Mislevy et al. 2002. 1.2. Generation of Verbal Items: Templates and Natural Language Generation It is not particularly surprising to note that applications of AIG have been concentrated in areas where the underlying cognitive domain is restricted in content and highly structured (e.g., various forms of abstract reasoning and mathematics.) Progress in automatic generation of verbal