Evaluating Questions in Context
Lee Alan Becker, Martha Stone Palmer, Sarel van Vuuren, Wayne Ward · 2011
We present an evaluation methodology and a system for rank-ing questions within the context of a multimodal tutorial dia-logue. Such a framework has applications for automatic ques-tion selection and generation in intelligent tutoring systems. To create this ranking system we manually author candidate questions for specific points in a dialogue and have raters as-sign scores to these questions. To explore the role of ques-tion type in scoring, we annotate dialogue turns with labels from the DISCUSS dialogue move taxonomy. Questions are ranked using a SVM-regression model trained with features extracted from the dialogue context, the candidate question, and the human ratings. Evaluation shows that our system’s rankings correlate with human judgments in question rank-ing. 1