Minimum description length criterion for modeling of chaotic attractors with multilayer perceptron networks
Yi Zhao, MICHAEL P. SMALL · IEEE Transactions on Circuits and Systems I Fundamental Theory and Applications · 2006
Overfitting has long been recognized as a problem endemic to models with a large number of parameters. The usual method of avoiding this problem in neural networks is to avoid fitting the data too precisely, and this technique cannot determine the exact model size directly. In this paper, we describe an alternative, information theoretic criterion to determine the number of neurons in the optimal model. When applied to the time series prediction problem we find that models which minimize the description length (DL) of the data, both generalize well and accurately capture the underlying dynamics. We illustrate our method with several computational and experimental examples.