Response to the Norris Commentary
DORRY MANN KENYON, Valerie A. Malabonga, Helen S. Carpenter · Language learning & technology · 2001
Norris has provided a thorough and insightful commentary to our article. Nevertheless, we would like to respond to four points that Norris has brought up. The first two points are clarifications about the COPI's design and our most recent research findings about this new test. The last two points respond to Norris' discussions of broader issues in the assessment of speaking, specifically the validity of the ACTFL Guidelines (1999) and the complexity of assessing second language speaking. First, a clarification about the adaptive algorithm of the COPI seems appropriate. In a lengthy performance-based assessment, efficiency is necessary. In the case of the COPI, raters need to listen to all responses made by examinees. With an essay prompt, in large-scale written tests, raters often evaluate in 1 to 2 minutes what examinees took 30 to 45 minutes to produce. With speaking assessments, however, if the examinee speaks for 20 minutes, the rater needs to listen for at least 20 minutes, assuming the rater listens only once. While a typical full-length Simulated Oral Proficiency Instrument (SOPI) presents 15 tasks to the examinee, the first 7 tasks may be administered as a short form for examinees at ACTFL Intermediate and Advanced levels. These 7 tasks comprise 4 tasks at the intermediate level and 3 at the advanced level. Our experience indicates that this approach produces an adequate speech sample to make for raters to arrive at their ratings (e.g., Kenyon & Tschirner, 2000). Based on this experience and to improve efficiency, we decided a priori that the COPI algorithm should also present examinees with 7 tasks, 4 at their starting level (i.e., the level of the self assessment) and 3 at the next level above. If the starting level was superior, then four tasks at the superior level and three at the advanced level were presented. The only instances in which more than seven tasks were administered to examinees occurred when, during the course of the COPI, they chose to be administered a task below their starting level or two levels above their starting level. The algorithm used in the COPI continued testing until at least four tasks at the starting level and three at the next higher level were administered. In a small number of cases (11 out of 54, or 20%), more than seven tasks were administered before this criterion was reached. One of the main goals in this approach was to ensure that examinees were not disadvantaged with a lower rating if they started at too low a level. Subsequent analysis of the examinees' COPI and SOPI scores, as reported in Kenyon, Malabonga, and Carpenter (2001), revealed that examinees' starting COPI level (based on their self-assessment) was problematic only for a small percentage (8 %) of the examinees. Second, we agree with Norris that there is still work to be done in researching the COPI. The current article reports on our first analyses of the COPI, focusing on the attitudinal questionnaires. Kenyon, Malabonga, and Carpenter (2001), for example, present further results on the relationship between examinee self-assessment of speaking proficiency, teacher assessment of examinee speaking proficiency, and COPI results. These analyses show that the student self-assessment instrument had a high rank-order correlation with examinees' actual performance on the COPI (.88), and that teacher assessment of student proficiency also correlated highly with the COPI (.84). Kenyon et al. also presented a preliminary analysis on examinee use of planning and response time, showing that more proficient examinees used less planning time but had longer response times. We intend to conduct further analyses of the COPI, including examining raters' behavior, investigating equivalencies of ratings across the three test formats and among tasks from the same ACTFL level, and conducting discourse analyses of examinee speech under the different testing conditions. Third, we feel it necessary to provide a broader perspective on Norris' comments regarding the ACTFL Guidelines (1999). …