Effect of Noise on Gradient Systems
Kevin I.-J. Ho, Hsia‐Ching Chang, Wei‐Bin Lee · International Journal of Computer Theory and Engineering · 2013
Recently, we have analyzed the convergence properties and derived the objective functions of the weight noise injection-based algorithms.In this paper, we generalize our work to perturbed gradient systems (PGS).For original gradient systems (GS), the change of a vector x(t) at time t is defined as the gradient vector of a function F(x) times a small negative constant.For PGS, it is assumed that the vector x(t) is perturbed by mean zero Gaussian noise.The noise can either be multiplicative or additive.In this paper, the corresponding energy functions for both cases are derived.It is found that their energy functions are very difference from F(x).For the case of multiplicative noise, the energy function consists of three terms: (i) F(x), (ii) a regularization term and (iii) a de-regularization term.For the case of additive noise, the energy function consists of only two terms: (i) F(x) and (ii) a regularization term.Note that de-regularization could lead to divergence behavior while regularization will improve the convergence behavior of a system.Our results suggest that special caution should be done to a gradient system with multiplicative noise.