Time series data mining for the Gaia variability analysis
K. Nienartowicz, Diego Ordóñez Blanco, L. P. Guy, B. Holl, Lecoeur-Taïbi, Isabelle, N. Mowlavï, L. Rimoldini, Ruiz, Idoia, Maria Süveges, L. Eyer · elib (German Aerospace Center) · 2014
Gaia is an ESA cornerstone mission, which was successfully launched December 2013 and commenced operations in July 2014. Within the Gaia Data Processing and Analysis consortium, Coordination Unit 7 (CU7) is responsible for the variability analysis of over a billion celestial sources and nearly 4 billion associated time series (photometric, spectrophotometric, and spectroscopic), encoding information in over 800 billion observations during the 5 years of the mission, resulting in a petabyte scale analytical problem. In this article, we briefly describe the solutions we developed to address the challenges of time series variability analysis: from the structure for a distributed data-oriented scientific collaboration to architectural choices and specific components used. Our approach is based on Open Source components with a distributed, partitioned database as the core to handle incrementally: ingestion, distributed processing, analysis, results and export in a constrained time window.