Is Question Answering Better than Information Retrieval? Towards a Task-Based Evaluation Framework for Question Series
Jimmy Lin · North American Chapter of the Association for Computational Linguistics · 2007
This paper introduces a novel evaluation framework for question series and employs it to explore the effectiveness of QA and IR systems at addressing users’ information needs. The framework is based on the notion of recall curves, which characterize the amount of relevant information contained within a fixed-length text segment. Although it is widely assumed that QA technology provides more efficient access to information than IR systems, our experiments show that a simple IR baseline is quite competitive. These results help us better understand the role of NLP technology in QA systems and suggest directions for future research.