ARTS: Assessing Readability & Text Simplicity
Björn Engelmann, Christin Katharina Kreutz, Fabian Haak, Philipp Schaer · 2024
Automatic text simplification aims to reduce a text's complexity.Its evaluation should quantify how easy it is to understand a text.Datasets with simplicity labels on text level are a prerequisite for developing such evaluation approaches.However, current publicly available datasets do not align with this, as they mainly treat text simplification as a relational concept ("How much simpler has this text gotten compared to the original version?") or assign discrete readability levels.This work alleviates the problem of Assessing Readability & Text Simplicity.We present ARTS, a method for language-independent construction of datasets for simplicity assessment.We propose using pairwise comparisons of texts in conjunction with an Elo algorithm to produce a simplicity ranking and simplicity scores.Additionally, we provide a high-quality human-labeled and three GPT-labeled simplicity datasets.Our results show a high correlation between human and LLM-based labels, allowing for an effective and cost-efficient way to construct large synthetic datasets.