General Solution Framework for Management and Monitoring of Cross-Platform Data Processes
V. Banovic, I. Soric · 2019
The purpose of this paper is a description of a proof-of-concept implementation for running and managing cross-platform data processes. The term “data process” in this context represents a variety of data management processes being executed by any of the common data integration tools and business intelligence processes ran through BI tools. Besides the common data management process workflow, custom data processing can be achieved by executing scenario-specific scripts, such as Big Data processing being achieved by running Spark or Hadoop jobs from inside the platform. The framework is implemented as a web application with a separate service, which handles job execution and scheduling. Besides that, the service is in charge of achieving parallelism, performance optimization and dependency management including the automatic discovery of different dependencies. The user interface offers monitoring of process execution through a graph representation of the execution plan or a regular table visualization. This paper explains the details of the developed concept and the architecture of our technology agnostic implementation.