Linear Regression with Regularized Hyperparameter Optimization
W. K. Li · Theoretical and Natural Science · 2025
Traditional linear regression struggles with noisy data due to its lack of regularization, which fails to mitigate overfitting effectively. This results in excellent fitting on training data but poor generalization on test sets. Additionally, its reliance on a single analytical solution via the normal equation limits adaptability to complex, dynamic data relationships, particularly under significant noise interference. To address these shortcomings, this paper introduces a linear regression method with regularized hyperparameter optimization (Reg-LR). Standard linear regression uses the normal equation to efficiently capture basic linear patterns, while Reg-LR incorporates hyperparameters to dynamically regulate model complexity, balancing fitting accuracy and generalization. By optimizing the loss function, this approach enhances performance from basic fitting to robust prediction. Experiments feature two key modules: a data generation module using NumPy to produce simulated datasets with Gaussian noise, simulating realistic conditions, and a regularization optimization module employing gradient descent to tune parameters across various hyperparameter values. Results indicate that standard linear regression achieves a test set mean squared error (MSE) of 3.90, while Reg-LR optimizes it to 6.56 through tuning. Though improvements are modest on small datasets, Reg-LR demonstrates robustness in noisy environments. Ablation studies highlight the regularization term’s role in preventing overfitting and the impact of hyperparameter choices on model stability. This method provides a scalable tuning framework for linear regression and a foundation for complex predictive tasks, offering theoretical and practical significance.