Automating Dependence-Aware Parallelization of Machine Learning Training on Distributed Shared Memory
Jinliang Wei, Garth A. Gibson, Phillip B. Gibbons, Eric P. Xing · 2019
Machine learning (ML) training is commonly parallelized using data parallelism. A fundamental limitation of data parallelism is that conflicting (concurrent) parameter accesses during ML training usually diminishes or even negates the benefits provided by additional parallel compute resources. Although it is possible to avoid conflicting parameter accesses by carefully scheduling the computation, existing systems rely on programmer manual parallelization and it remains a question when such parallelization is possible.