A Gentle Introduction to Stream Processing
Scott G. Haines · Apress eBooks · 2022
Learning to tackle and optimize data engineering problems can be challenging due to the many dimensions each problem can take on. At the outset of each new problem, you must think about data discovery, wrangling, ingestion, transformation, and data accountability, which is an umbrella relating to data contracts (strictly defined data definitions), as well as the need to optimize the data ingestion footprint (since data at scale can easily eat into operation costs). There are additional concerns relating to data access, lineage, and governance that need to be back of mind as well. Understanding how to use your collective knowledge to create quick plans of data attack is a skill that will get you far as a modern data engineer.