Ranking Passages for Argument Convincingness
Peter Potash, Adam Ferguson, Timothy J. Hazen · 2019
In data ranking applications, pairwise annotation is often more consistent than cardinal annotation for learning ranking models.We examine this in a case study on ranking text passages for argument convincingness.Our task is to choose text passages that provide the highest-quality, most-convincing arguments for opposing sides of a topic.Using data from a deployed system within the Bing search engine, we construct a pairwiselabeled dataset for argument convincingness that is substantially more comprehensive in topical coverage compared to existing public resources.We detail the process of extracting topical passages for queries submitted to a search engine, creating annotated sets of passages aligned to different stances on a topic, and assessing argument convincingness of passages using pairwise annotation.Using a state-of-the-art convincingness model, we evaluate several methods for using pairwiseannotated data examples to train models for ranking passages.Our results show pairwise training outperforms training that regresses to a target score for each passage.Our results also show a simple 'win-rate' score is a better regression target than the previously proposed page-rank target.Lastly, addressing the need to filter noisy crowd-sourced annotations when constructing a dataset, we show that filtering for transitivity within pairwise annotations is more effective than filtering based on annotation confidence measures for individual examples.