Move Real-Time Data Analytics to the Cloud: A Case Study on Heron to Dataflow Migration
Huijun Wu, Xiaoyao Qian, Aleks Shulman, Kanishk Karanawat, Tushar Singh, Hulya Pamukcu Crowell, Prashil Bhimani, Chunxu Tang, Yao Li, Lu Zhang, Chris Ulherr · 2021 IEEE International Conference on Big Data (Big Data) · 2021
Twitter is migrating its real-time data analytics infrastructure to the cloud. We propose a complete cloud migration procedure to tackle five identified challenges. Following the proposed solution, we successfully migrated a typical Heron job to Dataflow recently. This use case implements 1) control plane of provisioning and orchestration, 2) data plane of Heron to Beam programming model translation, 3) job IOs through replication and direct access/proxying, and 4) data validation by control test and metrics monitoring. The evaluation demonstrated the new Dataflow job matches the legacy Heron job in terms of performance and SLAs. Finally, we retrospect the migration process and identify future work on automated migration.