Fault tolerance training improves generalization and robustness
Robert Clay, Carlo H. Séquin · 2003
A recurrent theme in the neural network literature is that noise is good. Other researchers have presented experimental evidence of improvements due to adding noise to the input data, randomly presenting data rather than cycling through it, truncating bits of the weights, using ad hoc modifications of the error signal, stochastic updating, and others. Another source of noise, one that also forces the network to develop a more robust internal representation, is proposed. During training, one randomly introduces the types of failures that one might expect to occur during operation. It is shown how this leads to significant improvements in the network's ability to avoid the overfitting problem, generalize to new data, and cope with internal failures.>