PyrEval: An Automated Method for Summary Content Analysis
Yanjun Gao, Andrew Warner, Rebecca J. Passonneau · 2018
The pyramid method is a content analysis approach in automatic summarization evaluation for manual construction of a content model from reference summaries, and manual scoring of unseen summaries with the pyramid model.PyrEval automates the manual pyramid method.PyrEval uses low-dimension distributional semantics to represent phrase meanings, and a new algorithm, EDUA (Emergent Discovery of Units of Attraction), to solve a set cover problem to construct the content model from vectorized phrases.Because the vectors are pretrained, and EDUA is an efficient greedy algorithm, PyrEval can apply pyramid content evaluation with no retraining, and in excellent time.Moreover, PyrEval has been tested on many datasets derived from humans and machine generated summaries, and shown good performance on both.