Picky: Efficient and Reproducible Sharing of Large Datasets Using Merkle-Trees
Daniel Hintze, Andrew C. Rice · 2016
There is growing demand for researchers to share datasets in order to allow others to reproduce results or investigate new questions. The most common option is to simply deposit the data online in its entirety. However, this mechanism of distribution becomes impractical as the size of the dataset increases or if the dataset is frequently changing as new data is collected. In this paper we describe Picky, a new Merkle tree based system for sharing large datasets which allows users to download selected portions and to receive incremental updates. We demonstrate the viability of our approach by quantifying its benefit when applied to a number of large datasets used in the networking and measurement community.