Role of activation function on hidden units for sample recording in three-layer neural networks
Xingren Ying · 1990
It is shown that thekhidden units with asymptotic activation function are able to transfer any givenk+1 different inputs to linearly independent GHUVs (generated hidden unit vectors) by properly setting weights and thresholds. The number of hidden units with the LIT (linearly independent transformation) capability for the polynomial activation function is limited by the order of polynomials. For analytic asymptotic activation functions and given different inputs, the LIT is a generic capability and a probability 1 capability in setting weights and thresholds randomly. It is a generic and a probability 1 property for any random input if the weight and threshold setting has LIT capability for somek+1 inputs. For three-layer nets withkhidden units, in which the activation function is asymptotic and the output layer is without activation function, they are sufficient to recordk+1 arbitrary real samples. It is probability 0 to recordk+2 random real samples if the activation is a unit step function. This is true for the sigmoid function in the case of associative memory. These conclusions lead to a scheme for understanding associative memory in the three-layer networks