Dynamic Rate Control in the Kafka System
Michalis Tsenos, Nikos Zacheilas, Vana Kalogeraki · 2020
In this paper we propose RACER (Rate Adjusting ConsumER), a novel framework which implements a smart filtering and queuing mechanism that is able to deal effectively with sudden bursts and overloads which are frequently experienced in Big Data messaging systems. RACER provides rate control capabilities to the Kafka’s consumer API which allows us to effectively meet the requirements of the end-services without interference among them. Our experimental evaluation using smart city data, illustrates the benefits of our approach.