Tree Distance and Some Other Variants of Evalb
Martin Emms · 2008
Some alternatives to the standard evalb measures for parser evaluation are considered, principally the use of a tree-distance measure, which assigns a score to a linearity and ancestry respecting mapping between trees, in contrast to the evalb measures, which assign a score to a span preserving mapping.Analysis of the evalb measures suggests the other variants, concerning different normalisations, the portions of a tree compared and whether scores should be micro or macro averaged.The outputs of 6 parsing systems on Section 23 of the Penn Treebank were taken.It is shown that the ranking of the parsing systems varies as the alternative evaluation measures are used.For a fixed parsing system it is also shown that the ranking of parses from best-to-worst will vary according to whether the evalb or tree-distance measure is used.It is argued that the tree-distance measure ameliorates a problem that has been noted concerning over-penalisation of attachment errors.