Decentralized Storage for Scientific Data
Shirish Patel, Philip J. Rhodes · 2021 IEEE International Conference on Big Data (Big Data) · 2021
Emergence of new sensor technologies and increased parallelism has increased the size of scientific datasets dramatically in recent years. At the same time, research communities are demanding support for reproducible research, in which datasets and results are made available to other researchers. Such datasets do not change frequently, but future experiments can be derived from them.The volume of these datasets requires storage systems that scale across many machines, and also support highly selective queries to minimize the costs associated with unwanted data access.Decentralized Content Addressable Storage (CAS) systems such as the Inter-Planetary File System (IPFS) are promising as a way of disseminating datasets resulting from scientific research since they are scalable, the stored data is immutable, and versioning is straightforward. DS2, our prototype system for decentralized storage of spatial scientific data, bridges the gap between a scientist’s spatial view of a dataset and IPFS, the underlying decentralized filesystem. DS2 is intended to allow selective retrieval of data from very large datasets spanning many hosts.In this paper we investigate the feasibility of DS2 running on top of IPFS to store and retrieve large spatial scientific datasets. We also describe the decentralized metadata structures developed to support storage and retrieval of spatial datasets.