TRUST-TECH-Based Systematic Search for Multiple Local Optima in Deep Neural Nets
Zhiyong Hao, Hsiao‐Dong Chiang, Bin Wang · IEEE Transactions on Neural Networks and Learning Systems · 2021
Training deep neural networks (DNNs) rested heavily on efficient local solvers. Due to their local property, local solvers are sensitive to initialization and hyperparameters. In this article, a systematical method for finding multiple high-quality local optimal DNNs, based on the transformation under stability-retaining equilibria characterization (TRUST-TECH) method, is introduced. Our goal is to systematically search for multiple local optimal parameters for large models, such as DNNs, trained on large datasets. To achieve this, a dynamic searching path (DSP) method is proposed to provide improved search guidance used in TRUST-TECH. By integrating the DSP method with the TRUST-TECH (DSP-TT) method, multiple optimal training solutions with higher quality than randomly initialized ones can be obtained. To take advantage of these optimal solutions, a DSP-TT ensemble method is further developed. Experiments on various test cases show that the proposed DSP-TT method achieves considerable improvement over other ensemble methods developed for deep architectures. The DSP-TT ensemble method also shows diversity advantages over other ensemble methods.