Comparing Automated Machine Learning Against an Off-the-Shelf Pattern-Based Classifier in a Class Imbalance Problem: Predicting University Dropout
Leonardo Cañete-Sifuentes, Vı́ctor Robles, Ernestina Menasalvas, Raúl Monroy · IEEE Access · 2023
When facing a classification problem, data science practitioners must search through an armoury of methods. Often, practitioners are tempted to use off-the-shelf classifiers, including automated Machine Learning toolboxes; however, stand-alone classifiers are not applicable to any problem and automated Machine Learning may be time-consuming raising up environment-ethical issues. To magnify the problem, (commercial) automated Machine Learning toolboxes are black and practitioners are not allowed to extend them with new methods to improve their classification performance. In this paper, we present a case study, student dropout prediction, which most off-the-shelf classifiers find difficult to solve due to the problem’s inherent class imbalance. We shall see that MS Azure’s autoML outperforms a number of popular, stand-alone classifiers; yet, multivariate PBC4cip, an off-the-shelf classifier especially designed to deal with class imbalance, yields results that are just as good as MS Azure’s autoML. Our studies show that data science practitioners need to build themselves a taxonomy of classification mechanisms in terms of the properties of the problem to solve; additionally, autoML platforms should let scientists modify the armoury of classifiers and provide an explanation of both mechanism selection and mechanism tunning so that practitioners learn further lessons.