Effect of regularization term upon fault tolerant training
H. Takase, H. Kita, T. Hayashi · 2004
To enhance fault tolerance of multi-layer neural networks, we proposed PAWMA (partially adaptive weight minimization approach). This method minimizes not only output error but also the sum of squares of weights (the regularization term). This method aims to decrease the number of connections whose faults strongly degrade the performance of MLNs (important connections). On the other hand, weight decay, which aims to eliminate unimportant connections, is base on the same idea. This method expects to keeping important connections and decaying unimportant connections. In this paper, we discuss about the contradiction between two effects of the regularization term. Through some experiment, we show that the difference between two effects is brought by the partially application of the regularization term.