Communication-Efficient Weighted Reservoir Sampling from Fully Distributed Data Streams
Lorenz Hübschle-Schneider, Peter W. Sanders · 2020
We consider weighted random sampling from distributed data streams presented as a sequence of mini-batches of items. This is a natural model for distributed streaming computation, and our goal is to showcase its usefulness. We present and analyze a fully distributed, communication-efficient algorithm for weighted reservoir sampling in this model. An experimental evaluation on up to 256 nodes (5120 processors) shows good speedups, while theoretical analysis promises further scaling to much larger machines.