Using a Corpus of Sentence Orderings Defined by Many Experts to Evaluate Metrics of Coherence for Text Structuring
Nikiforos Karamanis, Chris Mellish · 2005
This paper addresses two previously unresolved issues in the automatic evaluation of Text Structuring (TS) in Natural Language Generation (NLG). First, we describe how to verify the generality of an existing collection of sentence orderings defined by one domain expert using data provided by additional experts. Second, a general evaluation methodology is outlined which investigates the previously unaddressed possibility that there may exist many optimal solutions for TS in the employed domain. This methodology is implemented in a set of experiments which identify the most promising candidate for TS among several metrics of coherence previously suggested in the literature. 1 1