Spoke 3 Big Data Management, Storage, and Archive Infrastructure
Giacomo Coran, Massimo Costantini, Sara Gelsumini, D. Busonero, C. Knapic, Andrea Adelfio, D. Ciangottini · 2025
The project aims to develop innovative data management solutions (database and file system archive) for the Spoke 3 division of the Italian National Center for HPC, Big Data, and Quantum Computing, rigorously applying the FAIR principles and Open Science methodologies. This system provides a robust and scalable framework designed to address big data challenges in astrophysics, which requires customized data models and management solutions. It focuses on managing observations from the Gaia and Fermi space missions, as well as simulations generated using the Pluto and Ramses codes, with the goal of supporting requests from the astrophysical Italian community on particular scientific cases.For Gaia data, we identified the HDF5 as a suitable format to improve data access and analysis outside the data reduction pipelines. The HDF5 format allows the nesting of primitive fields, arrays, and complex objects, thereby enhancing accessibility and computational efficiency. For the data management of the FT1 and FT2 data products from the Fermi Observatory, a common data model has been chosen to provide a unique point of access to the scientific data, employing logic for database queries and offering access to data products over a long observing period. Finally, to facilitate user interaction, a custom web application has been developed, providing a user-friendly interface for accessing and querying the archived metadata, additional “Cut & Merge” and “Transits” services, and supporting also secure authentication methods to ensure data privacy and compliance. These comprehensive resources enhance data accessibility and usability through the Spoke3 Archive Infrastructure, empowering the scientific community to explore and analyze astronomical data more effectively.