ClusterDataSplit: Exploring Challenging Clustering-Based Data Splits for Model Performance Evaluation
Hanna Wecker, Annemarie Friedrich, Heike Adel · 2020
This paper adds to the ongoing discussion in the natural language processing community on how to choose a good development set.Motivated by the real-life necessity of applying machine learning models to different data distributions, we propose a clustering-based data splitting algorithm.It creates development (or test) sets which are lexically different from the training data while ensuring similar label distributions.Hence, we are able to create challenging cross-validation evaluation setups while abstracting away from performance differences resulting from label distribution shifts between training and test data.In addition, we present a Python-based tool for analyzing and visualizing data split characteristics and model performance.We illustrate the workings and results of our approach using a sentiment analysis and a patent classification task.