Statistical physics theory of query learning by an ensemble of higher-order neural networks
Gustavo Deco, Dragan D. Obradovic · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · 1995
Query learning aims to improve the generalization ability of a network that continuously learns by actively selecting nonredundant data, i.e., data that contain new information about the process. In this paper, we formulate the problem of query learning in the statistical mechanical framework. We define an information theoretic measure of the informativeness of the newly presented data in order to decide if the latter should be used for the model update or not. Only the data that carry new information about the underlying process are selected for learning. The informativeness of the new data is defined as the Kullback-Leibler distance between the likelihood of the a posteriori parameter distributions obtained before and after the inclusion of the new data point. In order to make the problem analytically solvable, we formulate the theory for the ensemble of higher-order neural networks, i.e., for the case of polynomial models. Comparison with other theoretical approaches is included. Simulations that validate the proposed theory are also included.