M agpie : Improving the Efficiency of A/B Tests for Large Scale Video-on-Demand Systems
Hebin Yu, Haiping Wang, Chenfei Tian, Sandesh Dhawaskar Sathyanarayana, Shu Shi, Zhichen Xue, Shuaixin Yu, Haozhe Li, Yajie Peng, Xiaofei Pang, Rui-Xiao Zhang · 2024
With the exponential rise in video traffic, researchers and developers require more effective tools to validate the efficacy of designed algorithms for Video-on-Demand (VoD) system. However, traditional experimental platforms face two main challenges: a lack of realistic testing and the need for longer and significant effort. To overcome these limitations, we propose Magpie, an efficient experimental platform tailored for VoD systems. Magpie leverages a realistic operational setting, rapid testing, and high reproducibility to closely simulate online user environments without impacting production systems. Compared to conventional simulations, our evaluation demonstrates that Magpie reduces the disparity with online experiments by 85.6%. Deployed within our company-a leading video content provider in China-Magpie has efficiently validated over tens of algorithms, with 80% demonstrating enhanced performance in subsequent online tests.