From Data to Insights
Preethi S. H. Darius, Krishna Sowjanya K, V. N. Manju, Sanchari Saha, Paramita Mitra, Pratham Majumder, J Suneetha, Shreekanth M. Prabhu · 2023
Leading cloud service providers (CSPs) provide a plethora of services to manage big data. The goal is to ingest both structured and unstructured data through a pipeline and perform predictive analytics on the information. CSPs offer individual services for storage, processing, and analytics, and due to the repetitive nature of big data processing, these workflows are available as managed or open-source orchestration services. Stream and batch processing services are specialized for handling real-time unbounded stream data and large volumes of data that need to be processed batch-wise and stored in data lakes and warehouses. We analyze the services and data management patterns and pipelines from leading CSPs such as Amazon, Google, and Azure. The comparative study aims to provide various stakeholders of big data from data analysts to data scientists and novice researchers knowledge of the various options available and to aid in designing a cost-effective and efficient solution for any analytics application. The challenges and benefits of a big data solution are explored and a blueprint for choosing a right solution is outlined.