Towards a Better Understanding of Applied Textual Entailment: Annotation and Evaluation of the RTE-2 Dataset
Konstantina Garoufi · 2007
Applied textual entailment is a newly introduced generic empirical task that captures major semantic inferences across a wide spectrum of Natural Language Processing applications. In the present thesis we quest for a better understanding of the task by means of investigating a benchmark dataset for textual entailment, the dataset of the Second PASCAL Recognising Textual Entailment (RTE-2) Challenge. We propose a scheme for annotation of textual entailment, the Annotating RTE (ARTE) scheme, which models a range of diverse entailment mechanisms. The annotation of a considerable portion of the RTE-2 dataset following this scheme enables us an evaluation of the textual entailment data by gaining insights into the semantic-linguistic properties of the textual entailment phenomenon. Based on this evaluation, the thesis finally examines from various aspects the performance of the textual entailment systems participating in the RTE-2 Challenge, relative to different types of entailment. The methodology followed and the resulting observations make first steps towards a thorough analysis of systems’ performance, which is a key issue for the advancement of textual entailment technology.