The design and implementation of scalable data systems and incremental data sets
T. A. King, Steven P. Joy, Raymond J. Walker · AIP conference proceedings · 1993
Today’s space physics data systems are faced with a vast quantity of data and researchers need to have ready access to the data. These needs are compounded by the variety of ways in which researchers do their work. One way to provide ready access to data is through a central store. Even when there is an adequate central store of the data, researchers usually require local access to the data for performance reasons. Since the volume of data which may be distributed could be large it is very beneficial to distribute a data system with the data. Since system resources vary from researcher to researcher a data system must be scalable to the size of the resources on the system.Developers and researchers at the Planetary Plasma Interactions (PPI) Node of the Planetary Data System (PDS) have addressed this problem and have designed and implemented a scalable data system which uses incremental data sets. This system allows collections of data to be brought on‐line from archival media and merged with other collections in a seamless and easy manner. In addition, data sets may be migrated on and off line without the need to alter the access software. Access software is designed so that it can adapt to the current configuration of the data system. This allows independent evolutionary paths for the access software and the data sets.