Covariance-based weighting for optimal combination of model predictions
W.D. Penny · 1999
This paper introduces a method for calculating the covariance between different neural network solutions. It is based on a generalization of the delta method for calculating the network Hessian and generates what we call the `cross-covariance' matrix (its inverse is the `cross-Hessian'). Using this matrix we are able to estimate the covariance between network predictions at each point in input space, using training data alone. Whilst this is a significant result in itself we have also applied the method to the problem of finding optimal linear combinations of models. This results in a `covariance-based' weighted committee, where the weights are input-dependent. If the individual networks are unbiased then the covariance-based weighted committee is optimal in the sense of minimum expected prediction error.