Parser Evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics
Jennifer Foster, Josef van Genabith · 2008
We evaluate discriminative parse reranking and parser self-training on a new English test set using four versions of the Charniak parser and a variety of parser evaluation metrics.The new test set consists of 1,000 hand-corrected British National Corpus parse trees.We directly evaluate parser output using both the Parseval and the Leaf Ancestor metrics.We also convert the hand-corrected and parser output phrase structure trees to dependency trees using a state-of-the-art functional tag labeller and constituent-to-dependency conversion tool, and then calculate label accuracy, unlabelled attachment and labelled attachment scores over the dependency structures.We find that reranking leads to a performance improvement on the new test set (albeit a modest one).We find that self-training using BNC data leads to significantly better results.However, it is not clear how effective self-training is when the training material comes from the North American News Corpus.