Microsoft Research at RTE-2: Syntactic Contributions in the Entailment Task: an implementation
Lucy Vanderwende, Arul Menezes, Rion Snow · 2005
The data set made available by the PASCAL Rec-ognizing Textual Entailment Challenge provides a great opportunity to focus on the very difficult task of determining whether one sentence (the hypothe-sis, H) is entailed by another (the text, T). In RTE-1 (2005), we submitted an analysis of the test data with the purpose of isolating the set of T-H pairs whose categorization could be accu-rately predicted based solely on syntactic cues (Vanderwende and Dolan, 2005). Furthermore, the intent of our analysis was to isolate the impact of syntactic analysis in the limit, and not of any given parser. We therefore relied on human annotators to decide whether syntactic information from an idealized parser would be sufficient to make a judgment. We found that 34% of the test items could be handled by syntax, including basic alter-nations. We found that 48% of the test items could be handled by syntax plus a general purpose the-saurus. Given that the test data is split evenly be-tween entailments that are True and False, an accuracy of 74% is in principle achievable for a system with access to a general purpose thesaurus, if the system guesses randomly on what it cannot determine using syntax. With these numbers as our goal, we have developed MENT (