Distributed Resources for the Earth System Grid Advanced Management (DREAM)
Valerio Pascucci, Princeton Univ., Lawrence Livermore National Lab. (LLNL), Steve Petruzza · 2019
The DREAM project was funded more than 3 years ago to design and implement a next-generation ESGF (Earth System Grid Federation) architecture which would be suitable for managing and accessing data and services resources on a distributed and scalable environment. In particular, the project intended to focus on the computing and visualization capabilities of the stack, which at the time were rather primitive. At the beginning, the team had the general notion that a better ESGF architecture could be built by modularizing each component, and redefining its interaction with other components by defining and exposing a well defined API. Although this was still the high level principle that guided the work, the DREAM project was able to accomplish its goals by leveraging new practices in IT that started just about 3 or 4 years ago: the advent of containerization technologies (specifically, Docker), the development of frameworks to manage containers at scale (Docker Swarm and Kubernetes), and their application to the commercial Cloud. Thanks to these new technologies, DREAM was able to improve the ESGF architecture (including its computing and visualization services) to a level of deployability and scalability beyond the original expectations. Overall, during its three-year lifespan, DREAM has undertaken innovative research work and prototyping: many of its solutions (the new containerized architecture, the Visus visualization engine, the Solr Cloud based search service) were handed over to the ESGF collaboration, which will make them operational and use them as their future software basis.