Enhancing Inter-Service Communication Through Multi-Baseline Delta Encoding
Eduard Maltsev · 2024
In this study, we explore the potential of multi-baseline delta encoding (MDE) to enhance latency in distributed systems, specifically within the context of data-intensive inter-service communication using messaging brokers like Kafka. Unlike traditional delta encoding, which transmits differences between sequential data points, our approach is to transmit differences relative to baselines determined by time windows and unique identifiers. These baselines are stored in a shared registry, ensuring they are accessible to both producers and consumers for consistency and synchronization. In all distributed experiments, we use setups based on message brokers, and we aim to maintain the robustness and scalability of the messaging infrastructure while reducing the volume of transmitted data. We hypothesize that MDE can reduce message size by 20 % and overall latency by at least 10 %. Through a series of experiments and performance evaluations across diverse scenarios, we assess the efficacy of this approach in reducing latency and improving space efficiency. Our results are promising, showing a significant reduction in the serialized size of up to 44 % and median inter-service latency of up to 32 % compared to regular Avro encoding in certain scenarios.