Using Coh-Metrix to Access Cohesion and Difficulty in High-School Textbooks
David F. Dufty, Erin J. Lightman, Philip M. McCarthy, Danielle S. McNamara · eScholarship (California Digital Library) · 2006
Using Coh-Metrix to Assess Cohesion and Difficulty in High-School Textbooks Philip M. McCarthy, Erin J. Lightman, David F. Dufty, and Danielle S. McNamara Department of Psychology Memphis. TN 38152 {pmccarthy, elightman, d.dufty, d.mcnamara} @mail.psyc.memphis.edu) Table 1). The results confirmed out hypothesis: Cohesion indices were higher for science texts than for history texts (LSA, F(1, 273) = 437.72, p < .01; argument overlap, F(1, 273) = 742.07, p<.01). The FKGL difficulty index showed no significant difference between genres. Across chapters, our results suggested science texts were less cohesive near the end of units, whereas history texts tended to be more cohesive (see Table 1). Our study suggests that Coh-Metrix can facilitate sophisticated analysis of texts, helping to establish benchmarks and typical patterns of textual cohesion and difficulty. With greater understanding of cohesion between genres and across textual units, Coh-Metrix stands to offer a broader assessment of text that may better facilitate assignments of text to readers. Recent research in text processing has emphasized the importance of the cohesion of a text in comprehension (e.g., McNamara, 2001). Cohesion is the degree to which ideas in the text are explicitly related to each other and facilitate a unified situation model for the reader. Such research has led to the development of a computational tool, Coh-Metrix, (Graesser et al., 2004) that delivers over 300 indices of textual cohesion and difficulty. We hypothesized that a Coh-Metrix analysis of texts would indicate that cohesion indices - more so than traditional, shallow difficulty indices such as Flesch-Kincaid Grade Level (FKGL, Klare, 1974-75) - would identify characteristics of texts. Specifically, we hypothesized that within the expository domain, science texts would demonstrate greater cohesion than history texts, as the former dealt with less familiar subjects and would be likely to employ greater redundancy. We further hypothesized that as the parts of a text (beginning, middle, and end) serve different rhetorical purposes, that the sophisticated indices of Coh-Metrix would identify these differences. To test our hypothesis, we sampled three representative 1000-word sections from the beginning, middle and end of each chapter of seven commonly used high-school text books (three from science and four from history). Each section was analyzed using Coh-Metrix indices of Cohesion (argument overlap, latent semantic analysis (LSA), and number of connectives) as well as FKGL to assess difficulty. Acknowledgements This research was supported by the Institute for Education Sciences (IES R3056020018-02). References Graesser, A.C., McNamara, D., Louwerse, M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavioral Research Methods, Instruments, and Computers, 36, 193-202. Klare, G. R. (1974–1975). Assessing readability. Reading Research Quarterly, 10, 62-102. McNamara, D. S. (2001). Reading both high-coherence and low-coherence texts: Effects of text sequence and prior knowledge. Canadian Journal of Experimental Psychology, 55, 51-62. Results and Discussion We conducted an Analysis of Variance to assess differences between genres and across textual units (see Table 1. Results for Measures of Cohesion and Difficulty Science F-K LSA AO Con History Beginning Middle End Sig Beginning Middle End Notes: standard errors are in parentheses; * p<.05; ** p<.01 Sig