Let’s Measure Agreement, Consensus, and Polarization in Ordinal Data
Clem Aeppli, Didier Ruedin · 2022
There are different measures to capture agreement, consensus, concentration, dispersion, and polarization in ordinal data. To better understand how different measures work in practice, we compare consensus scores across specific situations: constructed cases, simulated data where we know the underlying distribution, and empirical data. Researchers have addressed the ‘problem’ of measuring agreement, consensus, and polarization in ordinal data in many ways across many subfields; we highlight the similarities as well as the distinctive features of these approaches. In many cases, the different measures are highly correlated, but the choice of method can lead to substantially different conclusions in situations that are difficult for non-mathematical users to predict. Unless researchers prioritize specific properties, we recommend a combination of measures and graphics to qualitatively examine the distribution.