Big data challenges, an insight into the GAIA Hadoop solution
Pierre-Marie Brunet · SpaceOps 2012 Conference · 2012
Large data sets, so-called big data, are no longer only a problem for web companies. Advances in data acquisition and telecommunications have created huge amounts of data. Storing, managing and processing this data are critical issues today. Gaia, an ESA mission for which CNES is a data processing center, will face the challenge of dealing with an end mission volume of one Petabyte stored in tables containing eighty billion rows. Unlike other big databases already in operation, the project involves complex data accesses and intensive computing. In this paper we will briefly present the studied products. Then we will describe the data management system based on Hadoop we have designed, which is a batch execution framework (MapReduce paradigm) with an underlying file system (HDFS) optimized for task execution. We will conclude by explaining how we have solved the many key questions raised by this conceptual migration: data access, workflow orchestration, designing a powerful Hadoop cluster taking into account its integration and the budget.