Hyper-parameter tuning through innovative designing to avoid over-fitting in machine learning modelling: a case study of small data sets
Muhammad Aftab, Tanvir Ahmad, Shahid Adeel, Sajjad Haider Bhatti, Muhammad Irfan · Journal of Statistical Computation and Simulation · 2025
Machine learning models play a vital role in prediction enhancement. The performance of these models highly depends on the appropriate selection of hyper-parameter values. Model architecture in addition, impacts the execution time to train and test a model. Values of the hyper-parameters are usually unknown and searching for optimal values is difficult. Hyper-parameters significantly affect the structure and the performance of a model, their setting is an important and interesting issue in machine learning algorithm manipulation. Several researchers have worked in this area; however, no one has investigated its relevancy to over-fitting. The present study has employed innovative response surface designs to tune hyper-parameters. These designs include a variety of experimental regions and are economical to explore optimal hyper-parameter values. Currently, less complex artificial neural network modelling has been employed. The study addresses the over-fitting problem by employing the least absolute shrinkage and selection operator regularization technique for limited observations.