Evaluation for Scenario Question Answering Systems
Matthew W. Bilotti, Eric Nyberg · 2006
Scenario Question Answering is a relatively new direction in Question Answering (QA) research that presents a number of challenges for evaluation.In this paper, we propose a comprehensive evaluation strategy for Scenario QA, including a methodology for building reusable test collections for Scenario QA and metrics for evaluating system performance over such test collections.Using this methodology, we have built a test collection, which we have made available for public download as a service to the research community.It is our hope that widespread availability of quality evaluation materials fuels research in new approaches to the Scenario QA task.This question begins with a statement about the general information need, then asks a yes or no question, and requests further information if the answer is yes.Finally, there is a follow-up question.What are the qualities that a good answer to this question should possess?