An Efficient Convex Training Algorithm for Artificial Neural Networks by Utilizing Piecewise Linear Approximations and Semi-Continuous Formulations
Ece S. K�ksal, Erdal Aydın, Metin T�rkay · Systems and Control Transactions · 2025
Artificial neural networks are widely used as data-driven models for capturing complex, nonlinear systems. However, suboptimal training remains a significant challenge due to the nonlinearity of activation functions and the reliance on local solvers, which makes achieving global solutions difficult. One solution involves reformulating activation functions as piecewise linear approximations to convexify the problem, though this approach often requires substantial CPU time. This study demonstrates that a tailored branch-and-bound algorithm can effectively address these challenges by efficiently navigating the solution space using linear relaxations. The proposed method achieves minimal training error, offering a robust solution to the training bottleneck. Unlike traditional mixed-integer programming approaches, which often struggle to converge within reasonable CPU times, the SOSX algorithm shows superior scalability, with computational demand growing almost linearly rather than exponentially as problem size increases. Experiments on datasets with the same dimensionality confirm that the algorithm's efficiency is not specific to a particular dataset, as it demonstrates consistent CPU time trends across multiple datasets.