Improving the Adam optimizer using time delays
Cayden Blake, Chase Ford, Benson Manner, Benjamin Z. Webb · Nonlinearity · 2025
Abstract One of the more popular optimization methods in current use is the Adam optimizer. This is due, at least in part, to its effectiveness as a training algorithm for deep neural networks, which are associated with many machine learning tasks. In this paper, we introduce time delays into the Adam optimizer. Time delays typically have an adverse effect on dynamical systems, including optimizers, slowing the system’s rate of convergence and potentially causing instabilities. However, our numerical experiments indicate that introducing time-delays into the Adam optimizer can significantly improve its performance, resulting in an often much smaller loss-value. Perhaps more surprising is that this improvement often scales with dimension-the higher the dimension the greater the advantage of using time delays in improving loss-values. Along with describing these results we show that, for the time-delays we consider, the temporal complexity of the delayed Adam optimizer remains the same as the undelayed optimizer and that the algorithm’s spatial complexity scales linearly in the length of the largest time-delay. Last, we extend the theory of intrinsic stability to give a criterion under which the minima, either local or global, associated with the delayed Adam optimizer are stable.