On a Scalable Entropic Breaching of the Overfitting Barrier for Small Data Problems in Machine Learning
Illia Horenko · Neural Computation · 2020
Overfitting and treatment of small data are among the most challenging problems in machine learning (ML), when a relatively small data statistics size [Formula: see text] is not enough to provide a robust ML fit for a relatively large data feature dimension [Formula: see text]. Deploying a massively parallel ML analysis of generic classification problems for different [Formula: see text] and [Formula: see text], we demonstrate the existence of statistically significant linear overfitting barriers for common ML methods. The results reveal that for a robust classification of bioinformatics-motivated generic problems with the long short-term memory deep learning classifier (LSTM), one needs in the best case a statistics [Formula: see text] that is at least 13.8 times larger than the feature dimension [Formula: see text]. We show that this overfitting barrier can be breached at a 10[Formula: see text] fraction of the computational cost by means of the entropy-optimal scalable probabilistic approximations algorithm (eSPA), performing a joint solution of the entropy-optimal Bayesian network inference and feature space segmentation problems. Application of eSPA to experimental single cell RNA sequencing data exhibits a 30-fold classification performance boost when compared to standard bioinformatics tools and a 7-fold boost when compared to the deep learning LSTM classifier.