Rubin's hybrid on premises-cloud data access center
William O’Mullane, Yusra AlSayyad, James Chiang, R. Dubois, Frossie Economou, Fabio Hernandez, Flora Huang, Tim Jenness, Kian-Tat Lim, Yee-Ting Li, Fritz Mueller, Dan Speck, Stephen Pietrowicz, Wei Yang · 2024
Cloud computing offers unparalleled flexibility, a constantly increasing set of “Infrastructure as a Service’’ capabilities, resource elasticity and security isolation. One of the most significant barriers in astronomy to wholesale adoption of cloud infrastructures is the cost for hot storage of large datasets - particularly for Rubin, a Big Data project sized at 0.5 Exabytes (500 Petabytes) over the duration of its ten-year mission. We are planning to reconcile this with a “hybrid” model where user-facing services are deployed on Google Cloud with the majority of data holdings residing in our on-premises Data Facility at SLAC. We discuss the opportunities, status, risks, and technical challenges of this approach.