Kafka-based Architecture in Building Data Lakes for Real-time Data Streams
Kiran Peddireddy · International Journal of Computer Applications · 2023
The purpose of this paper is to investigate how Kafka can be used to construct data lakes for real-time data processing.Kafka has gained widespread popularity as a data ingestion and processing tool that offers scalability, fault tolerance, and flexibility.The benefits of utilizing Kafka in a data lake architecture are analyzed, as well as the procedures involved in utilizing Kafka in a data lake architecture.In addition, a case study is provided of a major financial institution that utilized Kafka to establish a data lake.The significance of Kafka in modern data processing is emphasized in this paper, as well as its worth in developing data lakes for real-time data processing.