Efficient data storage of astronomical data using HDF5 and PEC compression
J. Portell, Enrique García-Berro, C. Estepa, J. Castañeda, Marcial Clotet · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 2011
Future space missions are based on a new generation of instruments that often generate vast amounts of data. Transferring this data to ground, and once there, between different computing facilities is not an easy task whatsoever. A clear example of these missions is Gaia, a space astrometry mission of ESA. To carry out the data reduction tasks on ground, an international consortium has been set up. Among its tasks perhaps the most demanding one is the Intermediate Data Updating, which will have to repeatedly re-process nearly 100 TB of raw data received from the satellite using the latest instrument calibrations available. On the other hand, one of the best data compression solutions is the Prediction Error Coder, a highly optimized entropy coder that performs very well with data following realitic statistics. Regarding file formats, HDF5 provides a completely indexed, easily customizable file with a quick and parallel access. Moreover, HDF5 has a friendly presentation format and multi-platform compatibility. Thus, it is a powerful environment to store data compressed using the above mentioned coder. Here we show the integration of both systems for the storage of Gaia raw data. However, this integration can be applied to the efficient storage of any kind of data. Moreover, we show that the file sizes obtained using this solution are similar to those obtained using other compression algorithms that require more computing power.