3. Priors for Neural Networks
Herbert K. H. Lee · Society for Industrial and Applied Mathematics eBooks · 2004
One of the key decisions in a Bayesian analysis is the choice of prior. The idea is that one's prior should reflect one's current beliefs (either from previous data or from purely subjective sources) about the parameters before one has observed the data. This task turns out to be rather difficult for a neural network, because in most cases the parameters have no interpretable meaning, merely being coefficients in a nonstandard basis expansion (as described in Section 2.3). In certain special cases, the parameters do have intuitive meanings, as will be discussed in the next section. In general, however, the parameters are basically uninterpretable, and thus the idea of putting beliefs into one's prior is rather quixotic. The next two sections discuss several practical choices of priors. This is followed by a practical discussion of parameter estimation, a comparison of some of the priors in this chapter, and some theoretical results on asymptotic consistency.