Using latent semantic analysis to grade brief summaries: A study exploring texts at different academic levels
Ricardo Olmos, José A. León, Guillermo Jorge‐Botana, Inmaculada Escudero · Literary and Linguistic Computing · 2012
In this study, we propose an integrated method to automatically evaluate very brief summaries (around 50 words) using the computational tool latent semantic analysis (LSA). The method proposed is based on a regression equation calculated with a corpus of a 100 summaries (the training sample) and is validated on a different sample of summaries (validation sample). The equation incorporates two parameters extracted from LSA: (1) the semantic similarity of the summary, measured using the summary–expert summaries method and (2) the vector length. The study is based on a sample of 786 summaries by students at four academic levels. All of these students summarized either an expository or a narrative text; their summaries were then evaluated by four graders on a scale of 0–10. The results support three ideas. First, that incorporating both parameters into the method is more successful than the traditional cosine measure. The reliability of LSA for evaluating summaries rises >0.80 level for the expository text. Second, that LSA shows practically the same level of sensitivity as the human graders to the quality of the summaries at different academic levels. Third, that the method overcomes a serious limitation of LSA: its difficulties evaluating very brief texts.