"Back to the Byte": Towards Byte-oriented Semantics for Streaming Storage
Raúl Gracia-Tinedo, Flavio Junqueira, Tom Kaitchuck · 2024
Event streaming systems, such as Apache Kafka, are increasingly used for data-intensive applications due to the proliferation of streaming sources of data and the requirement of responding with low latency. These systems center their internal design and external APIs around the concept of events. In this paper, we challenge the event-centric design as the primary design option for event streaming systems. Instead, we describe a streaming storage design based on a byte-oriented streaming primitive that supports key properties such as atomicity, conditional writes, and durability. Such a primitive allows us to implement not only a traditional event API, but also other APIs supporting, e.g., byte streams, key-values, and state synchronization. A byte stream API is relevant for storage workloads requiring large data transfers or low-latency byte buffer IO. We implement this concept in Pravega: a tiered storage system for data streams. Via experiments on AWS, we show the practicality of exposing byte APIs for storage workloads compared to events.