The LogAnswer Project at CLEF 2009
Ingo Glöckner, Björn Pelzer · CLEF (Working Notes) · 2009
The LogAnswer system, a research prototype of a question answering (QA) system for German, participates in QA@CLEF for the second time. The ResPubliQA task was chosen for evaluating the results of the general consolidation of the system and improvements concerning robustness and processing of administrative language. LogAnswer uses a machine learning (ML) approach based on rank-optimizing decision trees for integrating logic-based and shallow (lexical) validation features. The paragraph with the highest rank is then chosen as the answer to the question. For ResPubliQA, LogAnswer was adjusted to specics of administrative documents, as found in the JRC Acquis corpus. In order to account for the low parsing rate for administrative texts, indexing, answer type recognition, and all validation features were extended to sentences with a failed parse. Moreover, support for questions that ask for a purpose, reason, or procedure was added. Compared to the rst prototype of LogAnswer that participated in QA@CLEF 2008, there were no major changes in the resources employed. We have utilized the Eurovoc thesaurus for extracting denitions of abbreviations and acronyms but this knowledge was not activated by the questions in the ResPubliQA test set. Two runs were submitted to ResPubliQA: The rst run was obtained from the standard conguration of LogAnswer with full logic-based processing of results, while the second run was run with the prover switched o. It simulates the performance of the system when all retrieved passages have a failed parse. The results obtained for the two runs were almost identical. Given that our parser for German has generated a useful logical representation for less than 30% of the sentences in the JRC Acquis corpus, it is not surprising that logical processing had a minor eect. A systematic analysis of the results of LogAnswer for the dierent question categories revealed an unfavorable decision in the processing of denition questions that will now be xed. Moreover, questions asking for a procedure proved dicult