POSTER: Pattern-Aware Sparse Communication for Scalable Recommendation Model Training
Jiaao He, Shengqi Chen, Jidong Zhai · 2024
Recommendation models are an important category of deep learning models whose size is growing enormous. They consist of a sparse part with TBs of memory footprint and a dense part that demands PFLOPs of computing capability to train. Unfortunately, the high sparse communication cost to re-organize data for different parallel strategies of the two parts impedes the scalability in training.