Research on Heterogeneous Data Transfer Method for Power Energy Management
Zhao Lin, Jiang Chi, Sun Jianqiao, Weifeng Wang, Bi Fengjuan · 2021
At present, the amount of data collected by electricity information acquisition system increases dramatically. For current data growth and business needs, based on the research results of Sqoop and DataX, which are cross-platform heterogeneous data migration tools based on big data platform, this paper proposes to adopt Spark technology to study the cross-platform heterogeneous data migration technology of big data acquisition platform (cloud platform) and Oracle database. Two example tests were conducted to verify the support and performance of the Spark migration tool for migration between data in different formats and the ability to read, parse, and support data in particular formats.