MLP's hidden-node saturations and insensitivity to initial weights in two classification benchmark problems: parity and two-spirals
Eiji Mizutani, Stuart E. Dreyfus · 2003
Two two-class classification benchmarks, the parity problem and the two-spiral problem, are very difficult to solve using a standard single-hidden-layer MLP when trained with an incremental gradient method (i.e., pattern-by-pattern-mode steepest-descent-type algorithm), often called backpropagation (BP) algorithm. We show that the learning capacity of such an incremental-mode MLP with a single hidden layer can be augmented dramatically by careful choice of learning rates with special attention to hidden-node saturation. In particular, using a modified squared error objective function, we shall demonstrate that an MLP with only four hidden nodes can consistently solve the seven-bit parity problem while simultaneously developing an "insensitivity" to parameters initialized in a certain small range. In the two-spiral problem, we show a single hidden-layer MLP optimized with an incremental gradient (or BP) algorithm tends to be attracted by a singular point and explain how to avoid it or solving the problem perfectly. We hope our finding can be further generalized to some other problems in the future.