Comparison of data file and storage configurations for efficient temporal access of satellite image data
Asheer K. Bachoo, F. van den Bergh, Albert Gazendam · 2009
Recent improvements in sensor technology, together with increases in data acquisition frequency, have resulted in a surge in satellite data volume. A single tile from the gridded MODIS products, spanning a region of interest of approximately 10° by 10°, is stored as an image containing close to six million pixels, with data in multiple spectral bands for each pixel. Time series analyses of sequences of such images in order to perform automated change detection is a topic of growing importance. Traditional storage formats store such a series of images as a sequence of individual files, with each file internally storing the pixels in their spatial order. To construct a time series of a single pixel through time using such traditional storage solutions would require accessing hundreds of very large files, resulting in significant overheads which limit high-throughput analyses. We aim to reduce this bottleneck by restructuring the storage scheme for typical satellite imagery as temporal sequences in order to reduce overheads and improve throughput. Four data structures (using the hierarchical data format (HDF5), network common data format (netCDF) and a native file system approach) are implemented and compared in a series of experimental read tests to determine which format is most appropriate for implementation in the CSIR Cluster Computing Centre's facilities.