Customized deduplication algorithm for multi-source power grid dispatch data
Rui Chen, Kaiwen Zheng, Chuanxian Ai, Gang Yao, Jinglong Chen, Yang Fu · 2023
This paper proposes a self defined algorithm for multi source power grid dispatching data, which aims to solve the problem of redundancy and duplication of dispatching data in power systems. Firstly, by extracting features from power grid dispatch data, data between different data sources is uniformly formatted and compressed to reduce storage space. Then, a partial deduplication strategy is used to deduplicate the data from each data source. Finally, the experiment was conducted using the indicators of “space compression rate and deduplication rate”. The experimental results show that this algorithm can effectively deduplicate multi-source power grid scheduling data, and has a high deduplication rate and spatial compression rate.