Reliability enhancements for high-availability systems using distributed event streaming platforms
Ana-Maria Dincă, Sabina-Daniela Axinte, Ioan C. Bacivarov, Gabriel Petrică · 2023
The effectiveness of a performant high-availability system can be measured by the transaction operations’ execution speed and reliability metrics. The performance and reliability of traditional database interactions can be improved through multiple techniques, including incorporating emerging technologies, such as distributed event streaming platforms, in the new or existent software infrastructure. This study is underlining the improvements that were obtained through Kafka integration with existing database operations by running a series of performance tests on two separate endpoints, that communicated with an independent software module. One endpoint used an asynchronous communication system, through event streams, and the other one used a traditional synchronous communication, through REST calls. Performance indicators such as throughput, error rate and execution time were extracted and analyzed, and it resulted that the data sent through Kafka had a 0% loss rate. Also, due to the numerous configuration options available, the reached throughput, using the asynchronous communication system, was more than 10 times higher. The study continued with an analysis of other event streaming platform vendors, with their particular features, disadvantages and other companies that use that specific technology. The final section is dedicated to a guide of best practices that ease the transition to an asynchronous messaging system between independent software modules, or just the addition of a new event streaming pipeline, and how to achieve performant and resilient communication, with configurable back-up mechanisms.