Visualizing weight dynamics in the N-2-N encoder
Raymond Lister · 2002
L. Kruglyak has proven that sets of weights exist so that multi-layer perceptrons can solve arbitrarily large N-2-N encoder problems. Kruglyak's static geometric construction is extended to give a way of visualizing weights dynamics during learning. This visualization provides insight as to why backpropagation has difficulty in finding suitable N-2-N encoder weights for N>8. The author argues that this insight has general consequences relating to the danger of utilizing intermediate activity values in hidden units, and to difficulties with finding solutions for tightly constrained (but not necessarily large) problems.>