Cloud Data Pipeline Automation Framework with Integrated Budget Management
Sumant Kulkarni · International Journal of Advanced Trends in Computer Science and Engineering · 2020
Today is the era of Big Data and Cloud Computing.Organizations, Universities and Governments are investing heavily in big data analysis which provide strategically important insights.The cloud platforms, with virtually infinite amount of on demand and scalable resources, are used to execute the required data transformations and pipelines.As the scope of the data analysis activities widen, organizations need a framework to schedule the data pipelines and achieve the objectives of Budget management, Data pipeline prioritization and Reuse encouragement.This paper presents a "Cloud Data Pipeline Automation Framework" with several checks and controls.The framework was evaluated on Google Cloud Platform and the results establish the effectiveness of the framework.By using this automation approach, organizations would realize the true benefits offered by the "pay-as-you-use" cloud platforms.