A two-phase submission algorithm using dynamic programming for Kafka consumer committing offset
Ruiguang Lv, Gang Wu, Zhi Li, Runlin Li, Mimi Zhang · 2023
As an important middleware of distributed computing, message queue enables us to address issues like decoupling applications, asynchronous communication, fluctuating data traffic, and others. Kafka is a popular open-source message queue for collecting, processing, storing, and analyzing messages at scale, which is famous for its excellent performance, low latency, fault tolerance, and high throughput. In some scenarios like electronic trading, reliable consumption is the utmost concern. However, the native consumer client provided by Kafka cannot guarantee that messages are consumed successfully at least once. This paper proposes a two-phase submission algorithm using dynamic programming which can improve reliability of message consuming. This algorithm utilizes a two-phase submission to ensure that every message is consumed successfully at least once, and uses dynamic programming to improve submission efficiency. We conduct experiments to demonstrate that this algorithm is more efficient in contrast with the original submission strategy. In experiments, this algorithm must make sure that every message is consumed successfully at least once while various consuming failure rates are mocked.