Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts
Yifan Zhang, Yifan Luo, Yuan Yang, Andrew Chi-Chih Yao · 2025
We present Autonomous Data Selection (AutoDS), a method that leverages base language models themselves as zero-shot "generative classifiers" to automatically curate highquality mathematical texts.Unlike prior approaches that require human annotations or training a dedicated data filter, AutoDS relies solely on a model's logits to determine whether a given passage is mathematically informative and educational.By integrating Au-toDS into a continual pretraining pipeline, we substantially boost downstream performance on challenging math benchmarks (MATH, GSM8K, and BBH) while using far fewer tokens than previous methods.Empirically, our approach achieves roughly a twofold improvement in pretraining token efficiency over strong baselines, underscoring the potential of selfdirected data selection in enhancing mathematical reasoning.We release our curated AutoMathText dataset to facilitate future research in automated domain-specific data curation § .The AutoMathText dataset is available at https://huggingface.co/datasets/math- ai/AutoMathText.