Exploiting Structured Feature and Runtime Isolation for High-Performant Recommendation Serving
Xin You, Hailong Yang, Siqi Wang, Tao Peng, Chen Ding, Xinyuan Li, Bangduo Chen, Zhongzhi Luan, Tongxuan Liu, Yong Li, Depei Qian · IEEE Transactions on Computers · 2024
Recommendation serving with deep learning models is one of the most valuable services of modern E-commerce companies. In production, to accommodate billions of recommendation queries with stringent service level agreements, high-performant recommendation serving systems play an essential role in meeting such daunting demand. Unfortunately, existing model serving frameworks fail to achieve efficient serving due to unique challenges such as 1) the input format mismatch between service needs and the model's ability and 2) heavy software contentions to concurrently execute the constrained operations. To address the above challenges, we proposeRecServe, a high-performant serving system for recommendation with the optimized design ofstructured featuresandSessionGroupsfor recommendation serving. Withstructured features,RecServepacks single-user-multiple-candidates inputs by semi-automatically transforming computation graphs with annotated input tensors, which can significantly reduce redundant network transmission, data movements, and useless computations. Withsession group,RecServefurther adopts resource isolations for multiple compute streams and cost-aware operator scheduler with critical-path-based schedule policy to enable concurrent kernel execution, further improving serving throughput. The experiment results demonstrate thatRecServecan achieve maximum performance speedups of 12.3$\boldsymbol{\times}$and$22.0\boldsymbol{\times}$compared to the state-of-the-art serving system on CPU and GPU platforms, respectively.