Non-relational multi-level caching for mitigation of staleness & stragglers in distributed deep learning
Aswathy Ravikumar · 2021
For efficient distributed deep neural network design, mitigation of stale gradients and stragglers is necessary. The stale gradient problem occurs during the distribution and parallelism of the deep neural networks on the multi-cluster/nodes. The proposed solution for stragglers is to use distributed non-relational database to update the intermediate results of weights and their respective nodes. The results from the database are given to the parameter server. If any delay in the parameter data due to straggling is detected, immediately the straggled data will be configured in another node as a server-less function. In this approach, each node is equipped with distributed in-memory cache and a non-relational database at the parameter server. The parameter server node is an intelligent node working based on a runtime threshold set and the error analysis for fixing the optimal value of 'K' for the K-SGD. The proposed solution for stale data is to efficiently utilize the multiple GPU with multiple levels of caching in the cloud for better performance and reduced response time. Response time is reduced by offloading and pushing data close to the nodes in multiple levels of distributed cache. Using GPU cache and Elastic Cache, data is updated to the individual nodes in optimal time intervals. Thus, an integrated solution for stragglers and staleness in both data-parallel and model parallel distributed deep learning networks.