Scheduling of operations in big data processing center for Gaia mission
Chantal Panem, A. Jean-Antoine-Piccolo, Frederic Pailler · 2018 SpaceOps Conference · 2018
ESA's Gaia astronomy mission, launched on December 19th 2013, will provide the most accurate map of the Galaxy to date, observing more than one billion celestial objects in the Milky Way and beyond.The Gaia data processing is handled by a European Scientific Consortium: DPAC, composed of around 450 scientists and engineers of 20 countries in Europe, and six DPC (Data Processing Centres).CNES hosts one processing centre (DPCC for DPC CNES) that runs eight scientific chains, each of them using results from others chains running in several DPC.The Gaia operation plan is drawn on a cyclic processing of all the data acquired since the beginning of the mission.With more and more chains in the loop at each cycle, any delay in one centre impacts the other ones, so schedules have to be carefully monitored at any step.After a brief summary of the mission status and DPAC organization, the paper will describe the DPCC 'big data' Hadoop architecture dealing with challenging performance constraints.It will then focus on the preparation, scheduling and monitoring of the operations at DPAC level, then zooming inside DPCC timeline, with an overview of tools for resource sharing analysis.