State-of-the-art Multivariate Regression with a General Nk Hidden Multi-Layer Feed Forward Neural Network Model
Nonvikan Karl-Augustt Alahassa · 2021
We have introduced a novel multivariate regression model, called General NkHidden Multi-Layer Feed Forward Neural Network Model. The model prediction power comes essentially from a combination of Universal Approximation theorem, the Stochastic Gradient Descent convergence, and finally the use of ADAM optimizer which is also (at least) locally convergent. This Nk-architecture is proposed to fit any multivariate regression task, as well as any classification task when a softmax activation gate function is applied to the output layer nodes. The model can easily be augmented to thousands of possible layers without loss of predictive power, and has the potential to overcome our difficulties simultaneously in building a model that has a good fit on the test data, and don’t overfit. Its hyper-parameters, the learning rate, the batch size, the number of training times (epochs), the size of each layer, the number of hidden layers, all can be chosen experimentally with cross-validation methods. We have run some experiments with the Mulan Project Datasets [29] to illustrate the performance of the model against Random Forest with a number of estimators from 5 to 10, and a maximum depth from 10 to 30. Not only has the model surpasses the Random Forest model in all tested configurations, but, we have also found this General NkHidden Multi-layer Feed Forward Neural Network to be so effective as it reaches state-of-the-art performance for multivariate regression in terms of Mean Squared Error.