Linear independence of internal representations in multilayer perceptrons
Jagesh V. Shah, Chi‐Sang Poon · IEEE Transactions on Neural Networks · 1999
This investigation identifies the linear independence of the internal representation of the multilayer perceptron as an essential property for exact learning. The sigmoidal hidden unit activation function has the ability to produce linearly independent outputs. As a result, the minimum number of hidden units for a set of specified input is the number of patterns less the rank of the input patterns. In addition, the basis of many training algorithms is shown to inherently increase the number of linearly independent vectors in the internal representations, thereby increasing the likelihood of exact learning.