Mathematical Modeling and Prediction of Neural Network Training based on RC Circuits
Bhaskar Ghosh, Indira Kalyan Dutta, Albert H. Carlson, Michael Wayne Totaro · 2020
A recent study [1] and our experimental results show that a Neural Network(NN) often learns the most important features early during training. This results in accuracy during training to follow similar curves in each case. The magnitude of weight updates are the highest during the early iterations and weight updates stabilize with continued training of the NN. In this paper, we examine the training curve of a neural network with respect to an Iteration Weight value of τ. Using τ, we can predict the approximate maximum accuracy and the required number of training examples to reach that specific accuracy level. The model resembles the well understood Resistance-Capacitor(RC) Charging Circuit and appears to act accordingly. Our motivation stems from the fact that predicting the number of training iterations for a model to reach a desired level of accuracy has not been attained. As such, our work should be useful for researchers in their efforts to improve their training designs.