Using Semantic Overlap Scoring in Answering TREC Relationship Questions
Gregory A. Marton, Boris Katz · 2006
A first step in answering complex questions, such as those in the "Relationship" task of the Text REtrieval Conference's Question Answering track (TREC/QA), is finding passages likely to contain pieces of the answer-passage retrieval.We introduce semantic overlap scoring, a new passage retrieval algorithm that facilitates credit assignment for inexact matches between query and candidate answer.Our official submission ranked best among fully automatic systems, at 23% F-measure, while the best system, with manual input, reached 28%.We use our Nuggeteer tool to robustly evaluate each component of our Relationship system post hoc.Ablation studies show that semantic overlap scoring achieves significant performance improvements over a standard passage retrieval baseline.