Moonlight: A Push-based API for Tracking Data Lineage in Modern ETL processes
Marko Jamedzija, Zoran Duric · 2021
This paper presents Moonlight - a novel solution for tracking data lineage in Extract Transform Load (ETL) data processes. It gives a brief overview of the reasons behind the need for such a solution, yielding its expected requirements - the ability to easily store, update and view ETL metadata with a clear way of representing the data lineage. To comply with these requirements, Moonlight uses a push mechanism in the form of REST API to track the metadata (which contains the information about the inputs and outputs, pipeline owners and involved components, reasons behind transformations being made, etc.), but most importantly for providing the digestible presentation of all information in the form of JSON, Markdown and HTML output. Its main intuition is to be a tool that is easy to set up, use, and integrate, and that allows a better understanding of data flows and their faster debugging.