Near Real-Time Big Data Stream Processing Platform Using Cassandra
Gautam Pal, Gangmin Li, Katie M. Atkinson · 2018
Users are always impatient to get answers instantly from analytics system. If time to insight exceeds 10s of milliseconds, then the value is lost. Applications such as stock market, sensors, Twitter feed data or fraud detection can't afford to wait. This often means analyzing the inflow of data before it even stored to the database of records. Coupled with zero tolerance for data loss and the challenge gets even more daunting. In realtime Big Data scenario rather waiting for data to be collected as a whole at a long periodic interval, streaming analysis let us identify patterns and make informed decisions based on them-as data start arriving. When data are non-stationary, and patterns change with time, streaming systems adapt itself. This work describes near real-time data storage and processing approaches to analyze streams of data with respect to Cassandra NoSQL datastore. It provides an insight into optimizing Cassandra on a multi data center setup for near Real-Time Responses. The classic trade-off between low-latency and high-accuracy is conceptualized. The theoretical claims are corroborated with several thorough experimental analysis in Apache and Datastax distribution of Cassandra.