Supercomputing centers tight coupling to face Big Data Processing
Pierre-Marie Brunet, Jérome Gasperi, Maurice Poncet, Tristan Faure · SpaceOps 2016 Conference · 2016
The increasing number of satellites, the sensitivity of embedded sensors, performance of downstream links and new algorithms complexity has pushed data processing ground segment to the Big Data era.The amount of data volume and computing power required by current and upcoming missions are real challenges to anyone who has to design a data processing center.One common strategy which has a good performance/cost ratio is to split data or computation across multiple datacenters.However this is not highly effective in terms of data duplication and network bottleneck risks.The objective of our study is to go further to distribute data and computation on multiple sites with a global orchestration mechanism in order to optimize load balancing and minimize data transfers.This leads to a cluster of supercomputers where new challenges rise up.To be interoperable, supercomputers need to expose generic interfaces at different levels: data description, data transfers, processing description and processing execution.The objective of our study is to set up a federated data processing center for the benefit of Earth Observation in the scope of Copernicus Program.Scientists should access petabytes of products through a unique portal with the possibility of developing and pushing their own new algorithms without concern for federation complexity.In this paper we present namely the progress of our work.