Networks Using Combined Online/Batch Quasi-Newton Techniques
Hiroshi Ninomiya · 2012
This paper describes a robust training algorithm based on quasi- Newton process in which online and batch error functions are combined by a weighting coefficient parameter. The parameter is adjusted to ensure that the algorithm gradually changes from online to batch. Furthermore, an analogy between this algorithm and Langevin one is considered. Langevin algorithm is a gradient-based continuous optimization method incorporating Simulated Annealing concept. Neural network training is presented to demonstrate the validity of combined algorithm. The algorithm achieves more robust training and accurate generalization results than other quasi-Newton based training algorithms.