Correction to "Partial simultaneous updating in Hopfield memories"
Bruno Cernuschi-Frías · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · 2000
.a simple one. It happened that both were chosen to be linear functions and the resulting DTNNSuIOF is presented in Fig. 2. For the researchers from the robot control field it might be amusing to look at this totally linear (delay element is irrelevant) DTNNSuIOF as a proposal for the robot control. Precision of the order 10 in the training is guaranteed. (See the paper.) The appearance of the constant gains equaling 50 is also an intriguing one. The matrices and were not given in this example. Therefore, there is no comment on this part. At this point without any additional information, we may say that the whole algorithm failed. DTNNSuIOF cannot generalize. This is the real curse for any NN indeed. The author personally admitted that but, understandably, he put it much more softly: “The high accuracy of the testing results does not guarantee the high stage of neural network robustness, which will be investigated and discussed in the next paper. ” Few basic remarks are needed here. In the world of NN, both the test, i.e., validation phase and generalization properties are defined in terms of previously unseen inputs. However, there are no test results in the paper at all. The highly accurate results presented in Fig. 5 of the paper are the outputs of the DTNNSuIOF on training inputs. The fact that the very error during the training is of the order 10 does not say anything about the generalization properties of NN. In the NN field, the stories about the overtrained NN having high variance and low bias are very well known. (Here, the bias 0!) Let us paraphrase the author correctly—it is highly likely that the high accuracy of testing results does guarantee the high stage of neural network robustness, which does not have to be investigated and discussed in the next paper. One thing is obvious, the whole algorithm relies on and believes in overwhelming power of the (pseudo) inversion operation. The accuracy in training phase presented in the paper shows merely the accuracy of the supporting software in calculation of a matrix pseudo inverse. In highly nonlinear, high-dimensional and noisy environment, DTNNSuIOF approach as given in the paper cannot result with any useful NN design algorithm.