Learning hyperparameters for neural network models using Hamiltonian dynamics
Kiam Choo · Library and Archives Canada (Government of Canada) · 2000
Learning Hyperparameters for Neural Network Models Using Hamiltonian Dynamics Kiam Choo Master of Science Graduate Department of Computer Science University of Toronto 2000 We consider a feedforward neural network model with hyperparameters controlling groups of weights. Given some training data, the posterior distribution of the weights and the hyperparameters can be obtained by alternately updating the weights with hybrid Monte Carlo and sampling from the hyperparameters using Gibbs sampling. However, this method becomes slow for networks with large hidden layers. We address this problem by incorporating the hyperparameters into the hybrid Monte Carlo update. However, the region of state space under the posterior with large hyperparameters is huge and has low probability density, while the region with small hyperparameters is very small and very high density. As hybrid Monte Carlo inherently does not move well between such regions, we reparameterize the weights to make the two ...