Reasoning-based uncertainty estimation for scalable multidimensional media bias annotation: A benchmark across diverse media spaces
Yifan Liu, Aliya Maussymbayeva, Aslanbek Murzakhmetov, Alexandra Nemerenco, Yike Li, Dong Wang · Knowledge-Based Systems · 2025
Media bias significantly shapes public perception by reinforcing stereotypes and exacerbating societal divisions. Prior research has often focused on isolated media bias dimensions such as political bias or racial bias , neglecting the complex interrelationships among various bias dimensions across different topic domains. Moreover, models trained on existing media bias benchmarks exhibit substantial performance degradation when applied to recent social media content. This shortfall primarily arises because these benchmarks do not adequately reflect the rapidly evolving nature of social media content, which is characterized by shifting user behaviors and emerging trends. To this end, we introduce a new dataset collected from YouTube and Reddit over the past five years. To efficiently and reliably annotate our collected dataset, we propose U ncertainty E stimation-guided A nnotation R efinement ( UEAR ), a novel human-LLM collaborative annotation scheme. UEAR estimates the uncertainties of LLM-based annotations through a two-way reasoning process and refines uncertain annotations from LLM with human inputs. Our final dataset contains human-LLM joint annotations of YouTube and Reddit content across multiple bias dimensions (e.g., gender, racial) and diverse topic domains (e.g., politics, sports), capturing the complex interplay of biases across societal sectors. We evaluate the annotation performance of UEAR on a manually annotated test set, demonstrating that UEAR can reliably generate multidimensional bias identification annotations with a 0.9082 macro F1 score, with limited human refinement. Through our statistical analysis of the generated annotations, we identify significant differences in bias expression patterns and intra-domain bias correlations across different domains. The code and data are made publicly available.