Architectural Patterns for Data Pipelines in Digital Finance and Insurance Applications
Diego Burgos, Pavlos Kranas, Ricardo Jimenez-Peris, Juan Mahíllo · 2022
Abstract This chapter presents a holistic solution to the issue of data pipelining that ingest data as fast as needed, works with current and historic data, handles efficiently aggregates, and can handle them at any scale. This holistic solution minimizes the Total Cost of Ownership (TCO) of the storage systems needed to develop a data pipeline and minimizes the execution time of the data pipeline. In this direction, the chapter presents a range of architectural patterns for data pipelining and illustrates how the presented solution boosts their simplification and optimization.