University of Hagen at QA@CLEF 2006: Answer Validation Exercise
Ingo Glöckner · CLEF (Working Notes) · 2006
The paper features MAVE (MultiNet answer verification), a system for validating results of question-answering systems which serves as a testbed for robust knowledge processing based on multi-layered extended semantic networks (MultiNet [4]). The system utilizes logical inference rather than graph matching for recognising textual entailment, which makes it easily extensible by further knowledge encoded in logical axioms. In order to ensure robustness, the prover is embedded in a relaxation loop which subsequently skips ‘critical’ literals until a proof of the reduced query succeeds. MAVE uses the number of skipped literals as a robust indicator of logical entailment. Although the detection of ‘critical’ literals does not necessarily result in the minimal number of non-provable literals, the skipped literal indicator performed very well in the AVE experiments. MAVE parses the hypothesis strings and is therefore sensitive to syntactic errors in hypothesis generation. A regular correction grammar is applied to alleviate this problem. The system also parses the original question (when available) and tries to prove it from the snippet representation. The skipped-literal count for the question is then combined with a similarity index for hypothesis vs. found answer. This method boosts recall by up to 13%. Additional indicators are used for recognizing false positives. The filter increases precision by up to 14% without compromising recall of the system.