System-aware dynamic partitioning for batch and streaming workloads
Zoltán Zvara, Péter G. N. Szabó, Balázs Barnabás Lóránt, András A. Benczúr · 2021
When processing data streams with highly skewed and nonstationary key distributions, we often observe overloaded partitions when the hash partitioning fails to balance data correctly. To avoid slow tasks that delay the completion of the whole stage of computation, it is necessary to apply adaptive, on-the-fly partitioning that continuously recomputes an optimal partitioner, given the observed key distribution.