Research and Implementation of Complex Real-time Computing based on Data Middle Platform
Peng Wang, Jiajin Li, Baolong Xuan, Wei Lu, Xian Zhang, Jian Hu, Hailang Yan · 2023
With the development of big data, the need for realtime data processing becomes more urgent. However, the complexity of the big data business causes the current data components to not be well supported. For this reason, a complex real-time calculation method based on data center is proposed. Build incremental dimension tables and incremental fact tables based on the original micro-batch scheduling, and analyze the impact of business incremental data changes on statistical results. And through the analysis of business operations, simplify the data processing process, reduce the data Merge operation, and reduce the data processing time. Finally, the feasibility, correctness and real-time nature of the method are verified through the application of real-time calculation of electricity bills.