On Variability due to Local Minima and K-Fold Cross Validation
Caren Marzban, Jueyi Liu, Philippe E. Tissot · Artificial Intelligence for the Earth Systems · 2022
Abstract Resampling methods such as cross validation or bootstrap are often employed to estimate the uncertainty in a loss function due to sampling variability, usually for the purpose of model selection. In models that require nonlinear optimization, however, the existence of local minima in the loss function landscape introduces an additional source of variability that is confounded with sampling variability. In other words, some portion of the variability in the loss function across different resamples is due to local minima. Given that statistically sound model selection is based on an examination of variance, it is important to disentangle these two sources of variability. To that end, a methodology is developed for estimating each, specifically in the context of K-fold cross validation, and neural networks (NN) whose training leads to different local minima. Random effects models are used to estimate the two variance components—that due to sampling and that due to local minima. The results are examined as a function of the number of hidden nodes, and the variance of the initial weights, with the latter controlling the “depth” of local minima. The main goal of the methodology is to increase statistical power in model selection and/or model comparison. Using both simulated and realistic data, it is shown that the two sources of variability can be comparable, casting doubt on model selection methods that ignore the variability due to local minima. Furthermore, the methodology is sufficiently flexible so as to allow assessment of the effect of other/any NN parameters on variability.