Lineage Resource Manager
Sughosh V. Kaushik · Proceedings of the 2022 International Conference on Management of Data · 2022
Main memory columnar database systems such as HyPer [1], Hana [4], MemSQL [9] have rapidly grown in use for both OLTP and OLAP applications in enterprise software [5]. Lineage capture in such in-memory columnar database systems incurs memory and performance overhead[2]. Hence we want to control the lineage capture overhead in such systems. We propose a lineage resource manager that employs specific mechanisms to control the lineage capture overhead. For implementing the resource manager, we consider an in-memory columnar query execution engine that supports concurrent query execution, where operators have been instrumented to capture lineage. Controlling lineage capture overhead is more imminent in embedded analytical systems (database systems running alongside another process, not as a standalone process) like DuckDB [11] and MonetDBLite [10].