Deciphering Student Ideas on Thermodynamics Using Computerized Lexical Analysis of Student Writing
Luanna B. Prevost, Kevin C. Haudek, JOHN ERNEST MERRILL, Mark Urban‐Lurain · 2020
Abstract Deciphering student ideas on thermodynamics using computerized lexical analysis of student writingConstructed responses, in which students describe their understanding in their own language,provide better insight into their thinking than do multiple-choice assessments. However,constructed responses are not often employed in large enrollment courses due to the time andfinancial constraints involved in grading these assessments. In this study, we examined studentunderstanding of thermodynamics using computerized lexical analysis of constructed responsesin a large enrollment course (N=384). Students were asked to interpret a graph depicting changesin free energy during the course of a reaction using both multiple-choice and open-endedresponses. Opened ended responses were analyzed using SPSS Text Analytics for Surveys(TAFS). The software extracts scientific terms from the students’ writing and places them intocategories using custom dictionaries of science terms. We validated the automated lexicalanalysis by using the categories created by TAFS as independent variables in discriminantanalysis to predict expert scoring of the students’ writing. Our findings to date reveal i) thatstudents hold a mix of correct and incorrect ideas about thermodynamics, and ii) that this resultis masked by multiple-choice testing. Over 50% of the students answering multiple-choicecorrectly displayed incorrect, or both correct and incorrect conceptualizations in their open-ended responses. Our results support previous studies that have uncovered students’heterogeneous ideas about matter and energy conservation and acid-base chemistry using lexicalanalysis. These findings suggest that computerized lexical analysis can improve instructors’understanding of the heterogeneity of ideas that student harbor about key concepts in STEMdisciplines and inform assessment practices.