Deriving lexical and syntactic expectation-based measures for psycholinguistic modeling via incremental top-down parsing
Brian Roark, Asaf Bachrach, Carlos E. Cardenas, Christophe Pallier · 2009
A number of recent publications have made use of the incremental output of stochastic parsers to derive measures of high utility for psycholinguistic modeling, following the work of Hale (2001; 2003; 2006).In this paper, we present novel methods for calculating separate lexical and syntactic surprisal measures from a single incremental parser using a lexicalized PCFG.We also present an approximation to entropy measures that would otherwise be intractable to calculate for a grammar of that size.Empirical results demonstrate the utility of our methods in predicting human reading times.