Modeling human sentence processing data with a statistical parts-of-speech tagger
Jihyun Park · 2006
It has previously been assumed in the psycholinguistic literature that finite-state models of language are crucially limited in their explanatory power by the locality of the probability distribution and the narrow scope of information used by the model.We show that a simple computational model (a bigram part-of-speech tagger based on the design used by Corley and Crocker ( 2000)) makes correct predictions on processing difficulty observed in a wide range of empirical sentence processing data.We use two modes of evaluation: one that relies on comparison with a control sentence, paralleling practice in human studies; another that measures probability drop in the disambiguating region of the sentence.Both are surprisingly good indicators of the processing difficulty of garden-path sentences.The sentences tested are drawn from published sources and systematically explore five different types of ambiguity: previous studies have been narrower in scope and smaller in scale.We do not deny the limitations of finite-state models, but argue that our results show that their usefulness has been underestimated.