An Open-framework Big Data Analytic Platform Applied to Timeseries Sensor Data
Michael Balestrieri, Joshua Capuzzi, Kyle Chang, Hamed Valizadehhaghi · 2024
Emerging use-cases that involve vast amounts of high-resolution sensor data are prompting utilities to reconsider their conventional approaches to handling such data. In this paper, an open-platform solution is proposed for a big data analytics platform that leverages vendor-distributed software. The platform outlines a comprehensive solution to effectively handle large volumes of high-resolution sensor data, specifically targeting substation digital fault recorders. This open-platform approach is intended to be extensible to various other data streams and sensor types. To establish an end-to-end data pipeline from the edge to centralized databases, employed are a combination of tools and processes. This includes deploying an edge agent, MiNiFi, data ingestions using NiFi, and an extract-transform-load (ETL) process powered by Spark and Hive. Scalable device management is achieved through Cloudera Edge Management. Additionally, a database schema is introduced tailored to PoW data to ensure data integrity across multiple DFRs. Throughout the implementation of this approach, challenges included addressing high CPU usage stemming from the MiNiFi agent, configuring the Hadoop system, converting Spark code, and optimizing the ETL processing of continuous PoW files. By adopting an open-platform approach, adaptability of this platform provides insights into addressing key implementation challenges.