Mitigating Downstream Disruptions: A Future-Oriented Approach to Data Pipeline Dependency Management with the GCS File Dependency Monitor
Preyaa Atri · Journal of Artificial Intelligence Machine Learning and Data Science · 2023
This paper introduces the GCS File Dependency Monitor, a Python library designed to facilitate workflow management within data pipelines on Google Cloud Storage (GCS).The library addresses a common challenge: ensuring the timely arrival of dependent files before proceeding with subsequent data processing tasks.It achieves this by monitoring a designated GCS bucket for the presence of a specific file.If the file is not found within a user-defined timeframe, the library triggers configurable warning and error notifications via email.This paper delves into the functionalities, applications, and potential impact of the GCS File Dependency Monitor, highlighting its contributions to data pipeline efficiency and reliability.Additionally, the paper explores opportunities for further development, aiming to provide valuable insights for researchers and practitioners in the field of data engineering.