Master Thesis Communication Filters for Distributed Optimization with the Parameter Server
Yipei Wang · 2015
One of the bottlenecks for distributed machine learning system is the huge communication overhead. In this thesis, we explore the filtering idea to tackle this problem and narrow down the scope to study distributed optimization under parameter server framework. We first discuss the intuition in designing filters based on the redundancy hidden in optimization algorithms. We then derived convergence conditions in applying filters. Based on these analysis, we provide efficient algorithms using sampling and randomized rounding techniques. For applying filters in practice, we also discuss strategies in filtering jointly and the scalability influence. The filters are experimentally proved to be able to reduce communication cost significantly without affecting the learning performance.